AD-A209  932 


ARI  Research  Note  89-39 


#fC  Hit  (»*»’: 

Knowledge  Elicitation:  Phase  I  Final  Report 

Volume  I 


John  M.  Leddo,  Theresa  M.  Mullin,  Marvin  S.  Cohen,  Terry 
A.  Bresnick,  F.  Freeman  Marvin,  and  Mike  F.O’Connor 
Decision  Science  Consortium 


% 


for 


r 

f 


ARI  Field  Unit  at  Fort  Huachuca,  Arizona,  Georgia 

Julie  A.  Hopson,  Chief 


Systems  Research  Laboratory 
Robin  A.  Keesee,  Director 


June  1989 


DT1C 

ELECTE 
JUL  0  3  1989 


United  States  Army 

Research  Institute  for  the  Behavioral  and  Social  Sciences 


Approved  for  the  public  release;  distribution  is  unlimited 


89  7  03  101 


U.S.  ARMY  RESEARCH  INSTITUTE 

FOR  THE  BEHAVIORAL  AND  SOCIAL  SCIENCES 


A  Field  Operating  Agency  Under  the  Jurisdiction 
of  the  Deputy  Chief  of  Staff  for  Personnel 


EDGAR  M.  JOHNSON 
Technical  Director 


JON  W.  BLADES 
COL,  IN 
Commanding 


Technical  review  by 

Jon  J.  Fallesen 
Robert  M.  Hamm 


Accession  For 


NTIS  GRA&I 
DTIC  TAB 
Unannounced 
Justification. 


¥ 

□ 


By 

Distribut i or / 

Avail’d  1 1 ' : 

;•  f'odes 

Av;  11 

•  •  /or 

Dlst 

Sue  -j 

■;  al 

NOTICES 

DISTRIBUTION:  This  report  has  been  cleared  for  release  to  the  Defense  Technical  Information 
Center  (DTIC)  to  comply  with  regulatory  requirements.  It  has  been  given  no  primary 
distribution  other  than  to  DTIC  and  will  be  available  only  through  DTIC  or  the  National 
Technical  Informational  Service  (NTIS). 

FINAL  DISPOSITION:  This  report  may  be  destroyed  when  it  is  no  longer  needed.  Please  do  not 
return  it  to  the  U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences. 

NOTE:  The  views,  opinions,  and  findings  in  this  report  are  those  of  the  author(s)  and  should  not 
to  be  construed  as  an  official  Department  of  the  Army  position,  policy,  or  decision,  unless  so 
designated  by  other  authorized  documents. 


REPORT  DOCUMENTATION  PAGE 


la.  REPORT  SECURITY  CLASSIFICATION 

Unclassified 


2a.  SECURITY  CLASSIFICATION  AUTHORITY 


2b.  DECLASSIFICATION /DOWNGRADING  SCHEDULE 


4.  PERFORMING  ORGANIZATION  REPORT  NUMBER(S) 


Form  Approved 
OMB  No.  0704-01*8 


1b.  RESTRICTIVE  MARKINGS 


3  .  DISTRIBUTION /AVAILABILITY  OF  REPORT 

Approved  for  public  release; 
distribution  is  unlimited. 


S.  MONITORING  ORGANIZATION  REPORT  NUMBER(S) 

ARI  Research  Note  89-39 


6a.  NAME  OF  PERFORMING  ORGANIZATION 

Decision  Science  Consortium 


6c  ADDRESS  (City,  State,  end  ZIP  Code) 

1895  Preston  White  Drive,  Suite 
Res ton,  VA  22091 

300 

Ba.  NAME  OF  FUNDING  /SPONSORING 

8b.  OFFICE  SYMBOL 

ORGANIZATION 

(If  eppliceble) 

U.S.  Army  Research  Institute 

PERI-S 

8c  ADDRESS  (City,  State,  end  ZIP  Code) 

5001  Eisenhower  Avenue 

Alexandria,  VA  22333-5600 

7a.  NAME  OF  MONITORING  ORGANIZATION 

ARI  Field  Unit  at  Fort  Huachuca,  Arizona 


7b.  ADDRESS  (C/ty,  Start,  and  ZIP  Code) 

U.S.  Army  Intelligence  Center  and  School 
Fort  Huachuca,  AZ  85613-7000 


9.  PROCUREMENT  INSTRUMENT  IDENTIFICATION  NUMBER 


10.  SOURCE  OF  FUNDING  NUMBERS 


PROGRAM 
ELEMENT  NO. 


(WilVi 


1 1 .  TITLE  (Include  Security  Clessificstion) 

Knowledge  Elicitation:  Phase  I  Final  Report,  Volume  I 


vEllfeliR 


12.  PERSONAL  AUTHOR(S) 

J.M.  Leddo,  T.M. 


13a.  type  of  report 
Interim  Report 


16.  supplementary  notation 


11 3b.  TIME  COVERED 

14.  OATE  OF  REPORT  (Ye er.  Month.  Dey) 

1988.  Mav 

George  W.  Lawton,  contracting  officer's  representative 


17,  COSATI  CODES  I  18.  SUBJECT  TERMS  (Continue  on  reverse  if  necessery  and  identify  by  block  number) 

FIEL®  I  group  1  sub-group  |  Knowledge  Elicitation  Military  Intelligence 

Knowledge  Representation 


19.  ABSTRACT  (Continue  on  reverse  if  necessery  end  identify  by  block  number) 

""""^Thls  research  note  presents  a  framework  for  knowledge  elicitation  and  representation 
based  on  theories  of  how  experts  themselves  represent  knowledge.  Much  previous  work  on 
knowledge  elicitation  has  been  guided  by  requirements  of  the  systems  for  which  the 
knowledge  is  to  be  encoded. 

Two  methodologies  are  presented:  an  interpretive  method,  based  on  the  premise  that 
experts  organize  knowledge  in  a^top  down V fashion;  and  a  generative  method,  based  on 
the  premise  that  they  organize  knowledge  in  a  "bottom  up"  way. 

The  interpretive  representation  of  expert  knowledge  integrates  scripts,  object  frames, 
and  mental  models.  The  generative  representation  integrates  production  rules,  semantic 
nets,  and  mental  models.  Knowledge  elicitation  techniques  for  each  method  are  selected 
and  tailored  according  to  knowledge  requirements  of  these  representations. 

Both  methods  were  tested  between  subjects  on  ten  situation  development  specialists  and 
ten  order  of  battle  specialists  in  the  Army  Intelligence  domain.  Knowledge  models  ^,( OVER) 


20.  DISTRIBUTION /AVAILABILITY  OF  ABSTRACT  1 21  ABSTRACT  SECURITY  CLASSIFICATION 

G3 uNCLASsiriEo/uNUMiTEo  □  same  as  rpt.  O  dtic  users  1  Unclassified 


22a  NAME  OF  RESPONSIBLE  INDIVIDUAL 

George  W.  Lawton 


OO  Form  1473,  JUN  86 


22b  TELEPHONE  (Include  Are a  Code)  22c.  OFFICE  SYMBOL 

202/274-9420  PERI-SF 


Previous  editions  ere  obsolete. 


SECURITY  CLASSIFICATION  OF  THIS  PAGE 

UNCLASSIFIED 


UNCLASSIFIED 


1KCUWITV  CLASSIFICATION  OF  THIS  FAO»fW»i«w  Pf  MwO 


ARI  Research  Note  89-39 


19.  Abstract  (continued) 

were  constructed  and  evaluated  for  each  of  the  methods,  and  the  intelligence 
specialties.  Results  show  that  experts  use  a  variety  of  knowledge  structures 
in  processing  information  and  reasoning,  and  that  both  representations  are 
necessary  to  capture  expert  knowledge  adequately.  J- .  ,  j  _  h 


•kcunity  classification  of  this  FAOCfWA*"  Dme  attend) 


ACKNOWLEDGEMENTS 


The  authors  wish  to  thank  the  project's  COR,  Dr.  George  Lawton,  for  his 
help  and  support  throughout  the  first  year  of  the  project.  In  addition  to 
handling  administrative  matters,  Dr.  Lawton  participated  in  numerous 
"brainstorming"  sessions  and  contributed  to  the  development  of  many  of  the 
ideas  reported  here. 

We  would  also  like  to  thank  Drs.  Rex  Brown  and  Kathryn  Laskey  of  DSC,  for 
their  work  and  conceptual  contributions  to  the  project.  Thanks  is  due  to  Drs. 
Craig  Cook  and  Kenneth  Hayes  of  SST,  the  project's  subcontractor  for  their 
work  on  the  project.  A  special  thanks  is  due  to  Ennis  Whitehead  (MG,  U.S. 

Army  (Ret.)),  James  Hunt  (BG,  U.S.  Army  (Ret.)),  and  Ed  Thompson  (MG,  U.S. 

Army  (Ret.)),  of  BAL,  the  project's  subcontractor,  for  lending  their  superb 
military  expertise  and  their  assistance  in  the  data  collection  effort. 


KNOWLEDGE  ELICITATION:  PHASE  I  FINAL  REPORT 


EXECUTIVE  SUMMARY 


Research  Requirement: 

Much  of  the  previous  work  on  knowledge  elicitation  has  been  guided  by 
requirements  of  the  systems  for  which  the  elicited  knowledge  is  to  be  encoded. 
The  present  paper  reports  a  framework  for  knowledge  elicitation  and 
representation  based  on  theories  of  how  experts  themselves  represent 
knowledge . 

Procedure : 

Two  methodologies  are  presented:  an  interpretative  method,  based  on  the 
premise  that  experts  organize  knowledge  in  a  "top-down"  fashion,  3nd  a 
generative  method,  based  on  the  premise  that  experts  organize  knowledge  in  a 
"bottom-up"  fashion.  The  interpretative  representation  of  expert  knowledge 
integrates  scripts,  object  frames  and  mental  models.  The  generative 
representation  integrates  production  rules,  semantic  nets  and  mental  models. 
Knowledge  elicitation  techniques  for  each  method  are  selected  and  tailored 
according  to  knowledge  requirements  of  these  representations.  Both  the 
interpretative  and  generative  methods  were  applied  in  the  Army  intelligence 
domain,  between  subjects  on  ten  situation  development  specialists  and  ten 
order  of  battle  specialists.  Knowledge  models  were  constructed  and  evaluated 
for  each  of  these  two  methods  and  intelligence  specialities. 

Findings : 

Experts  use  a  variety  of  structures  in  representing  knowledge  and 
reasoning,  that  both  interpretative  and  generative  knowledge  structures  are 
necessary  to  capture  adequately  expert  knowledge.  In  addition,  the 
interpretative  method  seemed  best  at  eliciting  high-level  goal  and  planning 
knowledge,  while  the  generative  method  seemed  best  at  eliciting  situation 
specific  rules  and  content  knowledge.  Situation  development  specialists 
tended  to  have  more  high-level  and  integrative  knowledge  while  order  of  battle 
specialists  tended  to  have  more  content  knowledge. 

Utilization  of  Findings: 

The  findings  were  used  to  evaluate  the  strengths  and  weaknesses  of  the 
two  methods  plus  suggest  ways  for  developing  an  improved  integrated  method. 

In  addition,  the  findings  suggest  ways  in  which  knowledge  requirements  and 
problem  solving  processes  may  differ  across  functional  areas  such  as  situation 
development  and  order  of  battle  analysis. 
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1.0  INTRODUCTION 


1.1  Objectives 

In  the  past  decade  and  a  half,  increasing  interest  within  both 
cognitive  psychology  and  artificial  intelligence  has  been  focused  on 
methods  for  eliciting  and  representing  expert  knowledge  (e.g.,  Newell  and 
Simon,  1972;  Schank  and  Abelson,  1977;  Hayes-Roth  et  al.,  1983).  As  a 
practical  by-product,  computer-based  aids  (expert  systems)  have  been 
designed  to  advise  decision  makers  in  a  variety  of  fields,  such  as 
medicine  and  geological  exploration;  moreover,  there  is  growing  interest 
in  other  applications,  such  as  matching  personnel  to  jobs,  the  design  of 
training  programs,  and  the  utilization  of  expertise  to  develop  new 
standard  operating  procedures  or  doctrine.  Despite  its  importance, 
however,  knowledge  elicitation  continues  to  be  an  ill -understood  and 
eclectic  art,  which  demands  enormous  amounts  of  time  and  unusual  forms  of 
skill  from  both  the  elicitor  and  Che  expert.  The  elicitation  process,  and 
its  attendant  difficulties,  raise  fundamental  issues  about  the  data 
structures  and  reasoning  strategies  associated  with  diverse  types  of 
expertise,  and  about  the  manner  in  which  such  knowledge  is  tapped  by 
different  elicitation  methods.  These  issues  have  not  yet  been  dealt  with 
in  a  satisfactory  way,  either  in  the  artificial  intelligence  community  or 
by  cognitive  psychologists. 

The  primary  purpose  of  the  work  reported  here  is  to  amplify  the 
synergy  between  theory  and  practice  in  the  knowledge  engineering  field: 
in  particular,  to  exploit  theoretical  models,  findings,  and  methods  in 
cognitive  science,  in  order  to  develop  and  test  an  improved  methodology 
for  eliciting  and  representing  knowledge. 

The  present  report  summarizes  the  results  of  the  first  year 
(Phase  I)  of  a  two-phase  project  sponsored  by  the  U.S.  Army  Research 
Institute  (ARI) .  The  principle  products  of  the  completed  research  will 
include: 

•  a  set  of  documented,  improved  knowledge  elicitation  methods; 

•  a  documented,  effective  framework  for  representing  the  knowledge 
that  is  elicited; 

•  a  preliminary  (but  expandable)  set  of  guidelines  for  the  use  of 
different  representation  structures  and  elicitation  techniques 
as  a  function  of  the  domain,  the  task,  the  background  of  the 
expert,  etc.;  and 

•  the  design  of  a  computerized  knowledge  elicitation  tool  to 
assist  in  planning,  conduct,  and  post-analysis  of  knowledge 
elicitation  sessions.  The  aid  will  Implement  the  elicitation 
methods,  knowledge  structures,  and  guidelines  previously 
developed. 

This  report  addresses  our  results  to  date  under  all  but  the  last  of 
these  headings. 
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In  conjunction  with  our  sponsor,  we  have  selected  a  specific  area  of 
knowledge  as  a  concrete  focus  and  testbed.  The  domain  of  primary  interest 
is  tactife'i  military  intelligence  and  within  that,  the  subdomain  of 
intelligence  analysis.  We  have  focused  on  two  specific  tasks  at  the  Corps 
and  ’'ivision  level:  maintenance  of  an  enemy  order  of  battle  (i.e.,  the 
locations,  composition,  and  organizational  arrangement  of  enemy  forces) 
and  situation  development  (in  particular,  assessing  enemy  capabilities  and 
likely  courses  of  action).  Finally,  specific  scenarios  and  problems  have 
been  developed  within  those  tasks.  Despite  this  detail,  the  elicitation 
methodology  under  development  is  quite  general  across  scenarios,  problems, 
and  tasks  in  the  intelligence  domain.  It  is  also  intended  to  be 
applicable  to  domains  that  share  key  characteristics  with  intelligence 
analysis:  in  particular,  reasoning  with  incomplete,  unreliable,  and 

inconsistent  information. 

Hoped  for  improvements  in  the  elicitation  process  take  a  variety  of 
forms :  viz . , 


•  reduction  in  the  time  required  for  elicitation, 

•  increase  in  the  consistency,  accuracy,  and  generality  of  the 
knowledge  elicited,  and 

•  expansion  in  the  types  of  expertise  or  degrees  of  expertise 
that  are  subject  to  elicitation. 

While  the  primary  goal  of  the  project  is  to  develop  a 
general-purpose,  theory-based  elicitation  methodology,  it  will  produce 
secondary  products  that  are  of  immediate  use  to  the  Army:  (1)  a  partial 
knowledge  base  for  tactical  military  intelligence  analysis,  and  (2)  a 
computerized  elicitation  tool  that  incorporates  and  utilizes  built-in 
knowledge  of  the  tactical  military  intelligence  domain. 

1.2  Basic  Approach  and  Overview  of  the  Report 

The  starting  point  for  the  present  research  is  a  conception  of 
knowledge  elicitation  as  a  constructive  process:  i.e.,  that  knowledge 
elicitation  is,  at  the  most  general  level,  a  joint  problem-solving 
enterprise  by  the  expert  and  the  knowledge  engineer.  This  conception 
implies:  (a)  that  pre-existing  knowledge  is  not  simply  transferred  from 

the  expert  to  the  knowledge  engineer  and  from  the  knowledge  engineer  into 
the  computer,  but  is  "constructed"  (to  varying  degrees)  during  and  after 
the  elicitation  dialogue.  In  other  words,  the  elicitor's  queries  often  do 
not  trigger  a  canned  response;  in  many  instances  they  present  a  "problem" 
to  be  solved,  through  reasoning  as  well  as  retrieval,  by  the  expert; 
similarly,  the  expert's  answers  often  present  a  "problem"  (of 
interpretation,  follow-up,  and  formalization)  to  be  solved  by  the 
elicitor;  (b)  that  the  quantity  and  quality  of  the  knowledge  that  is 
elicited  thus  depend  on  the  coordinated  cognitive  processes  of  both  the 
expert  and  the  elicitor.  Section  2.0  outlines  the  basic  support  for  this 
concept  in  the  cognitive  science  literature. 
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At  a  more  specific  level,  however,  the  existing  cognitive  science 
literature  provides  little  guidance.  A  variety  of  competing  structures 
for  representing  knowledge  (e.g.,  scripts,  frames,  rules,  semantic 
networks,  and  mental  models)  have  been  proposed,  but  there  is  little 
systematic  understanding  of  their  comparative  strengths  and  weaknesses  for 
describing  different  types  of  expert  performance,  or  the  way  they  might  be 
related  to  one  another  in  an  integrated  knowledge  representation. 
Similarly,  a  variety  of  procedures  for  tapping  expert  knowledge  have  been 
utilized  in  practice  and  in  experimental  research  (e.g.,  think-aloud 
problem  solving,  recall,  sorting  and  similarity  judgments,  cloze 
(filling-in-the-gaps)  experiments,  etc.)  but  there  has  been  little  effort 
to  study  the  capabilities  and  limitations  of  the  procedures  themselves,  or 
to  incorporate  them  into  a  systematic,  integrated,  practical  method  of 
knowledge  elicitation. 

The  initial  task  of  Phase  I  involved  development  of  appropriate 
theory.  Existing  research  on  knowledge  elicitation  and  representation  was 
examined  (including  theories,  models,  and  research  findings,  e.g.,  Leddo, 
Cardie  and  Abelson,  1987);  a  set  of  candidate  elicitation  techniques  and 
representation  methods  were  developed;  and  hypotheses  were  generated 
regarding  their  appropriateness  for  different  types  of  problems  and 
experts.  A  specific  objective  was  to  develop  two  contrasting 
methodologies  for  comparison:  one  appropriate  for  data-driven,  bottom-up, 
or  generative ,  modes  of  expert  reasoning,  and  the  other  appropriate  for 
expectancy -driven,  top-down,  or  interpretative ,  modes  of  reasoning.  Each 
methodology  was  to  include  multiple  elicitation  and  representation 
techniques. 

The  principle  results  of  this  task  are  described  in  Sections  3.0 
through  5.0.  In  sum,  they  include: 

•  two  integrated  knowledge  representation  structures:  an 
interpretative  structure  that  systematically  combines  scripts, 
frames,  and  mental  models  in  complementary  roles,  and  a 
generative  structure  that  in  a  similar  way  combines  rules, 
semantic  relations,  and  mental  models  (Section  3.0); 

•  two  integrated  sets  of  elicitation  techniques:  an 
interpretative  approach  that  combines  a  top-down,  template -based 
interview  with  think-aloud  problem-solving  to  instantiate 
principles  elicited  in  the  interview,  and  a  generative  approach 
that  combines  highly  data-driven  problem-solving  with  a 
bottom-up  interview  to  extract  general  principles  implicit  in 
problem-solving  performance.  A  general  framework  has  been 
developed  that  characterizes  these  (and  other)  elicitation 
techniques  in  terms  of  the  cognitive  processes  required  of  the 
elicitor  and  the  expert  (Section  4.0); 

•  a  set  of  guidelines  for  the  use  of  generative  versus 
interpretative  approaches,  as  a  function  of  the  expert  (e.g., 
his  cognitive  style  and  organizational  level)  and  the  problem 
(e.g.,  degree  of  uncertainty)  (Section  5.0). 
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The  second  task  in  Phase  I  involved  a  demonstration  of  the 
feasibility  of  the  candidate  elicitation  and  representation  methodologies 
in  the  selected  Army  operational  testbed,  and  an  informal,  qualitative 
comparison  of  the  generative  and  interpretative  approaches  across  two  Army 
tactical  intelligence  tasks:  order  of  battle  and  situation  development. 
Testing  occurred  in  two  phases:  (1)  a  pilot  phase  (involving  retired 
intelligence  officers  provided  by  Burdeshav  Associates,  Limited,  and 
intelligence  specialists  at  Fort  Huachuca  and  Fort  Leavenworth),  and  (2)  a 
primary  phase  (involving  intelligence  specialists  from  the  82nd  Airborne 
Division  at  Fort  Bragg,  the  4th  Infantry  Division  at  Fort  Carson,  and 
instructors  at  the  Army  Intelligence  Center  and  School  at  Fort  Huachuca) . 
The  methods  and  rationale  of  these  two  phases  are  described  in  Section 
6.0.  Appendices  B  and  C  contain  materials  used  in  the  primary  phase  of 
the  demonstration. 

A  by-product  of  these  field  demonstrations  was  that  actual  knowledge 
relating  to  the  conduct  of  Army  intelligence  was  collected  and 
represented.  Since  the  primary  purpose  of  the  elicitation  was  to  develop 
a  better  understanding  of  the  techniques,  virtually  any  knowledge  relevant 
to  performance  in  the  selected  testbed  was  a  candidate  for  elicitation. 
Nevertheless,  wherever  it  was  consistent  with  the  primary  research 
objective,  attention  was  focused  on  knowledge  that  (a)  appeared  more 
critical  for  potential  technology  applications  (expert  systems,  personnel 
screening,  and  training  in  that  order),  and  that  (b)  is  not  adequately 
elicited  or  represented  by  existing  methods.  The  latter  category  includes 
knowledge  that  goes  beyond  written  doctrine  and  standard  training, 
knowledge  that  is  not  naturally  represented  by  rules,  knowledge  that  is 
not  easily  articulated,  knowledge  involved  in  exceptional  (expert-level) 
performance,  knowledge  of  a  rapidly  evolving  domain,  etc.  In  general,  our 
interest  extended  both  to  substantive  beliefs  about  the  domain  (e.g., 
enemy  tactics)  and  to  methods  of  reasoning  with  those  beliefs.  Appendix  D 
contains  four  formally  represented  databases  of  elicited  knowledge, 
derived  from  the  interpretative  and  generative  approaches,  and  pertaining 
to  order  of  battle  and  situation  development  respectively.  (We  should 
note,  however,  that  within  the  constraints  of  time  and  resources  imposed 
by  this  project,  it  is  impossible  to  guarantee  the  completeness  or 
accuracy  of  all  the  elicited  data.) 

The  final  task  in  Phase  I  involved  the  development  of 
representational  models  for  the  elicited  knowledge,  and  an  evaluation  of 
the  performance  of  the  two  methodologies  as  a  function  of  characteristics 
that  differentiate  the  two  Army  intelligence  tasks.  Appendix  D,  as  noted, 
presents  the  representational  models.  Section  7.0  Includes  annotated 
protocols  of  elicitation  sessions,  followed  by  step-by-step  descriptions 
of  the  process  of  building  a  knowledge  model  from  the  transcript  data. 
Section  8.0  presents  conclusions  of  the  first  year  of  research  and 
outlines  plans  for  Phase  II. 
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2.0  COGNITIVE  SCIENCE  BACKGROUND:  A  CONSTRUCTIVE 
VIEW  OF  KNOWLEDGE  ELICITATION 


2.1  The  Mvth  of  "Mining  for  Nuggets"  of  Knowledge 

The  central  question  in  the  field  of  artificial  intelligence, 
whether  computers  can  be  made  to  Chink,  has  always  focused  on  tasks  (such 
as  chess)  where  a  formal  algorithmic  solution  (i.e.,  an  automatic 
"mindless"  approach)  was  unavailable  or  impractical.  Early  work  in 
artificial  intelligence  focused  on  general-purpose  technologies  for 
reasoning.  For  example,  one  very  general  approach  to  problems  is 
means-ends  analysis  (Newell  and  Simon,  1972):  compare  the  goal  (e.g., 
checkmate)  with  the  existing  state  of  affairs  and  search  memory  for 
operations  (e.g.,  chess  moves)  which  will  reduce  the  difference  (this  may 
recursively  involve  searching  for  operations  to  reduce  differences  between 
a  state-of-affairs  and  the  condition  for  applying  the  operator) .  Methods 
such  as  this  were  implemented  in  simple,  well-defined  contexts  besides 
chess,  such  as  blocks-world  (Winograd,  1972)  and  "cryptarithmetic"  (Newell 
and  Simon,  1972) .  The  focus  of  research  shifted  to  expert  systems  with 
the  realization  that  such  general-purpose  techniques  were  not  sufficient 
for  the  solution  of  more  realistic,  complex  problems. 

In  many  such  problems  (including  chess),  the  number  of  possible 
solutions  (i.e.,  combinations  of  operations  that  might  link  the  goal  and 
the  existing  state  of  affairs)  is  very  large.  In  those  cases,  real-time 
outputs  depend  on  methods  for  reducing  search  effort- -e . g. ,  the  selection 
of  representation  schemes,  or  ways  of  slicing  up  and  describing  the 
problem,  which  are  highly  tailored  to  a  specific  domain  and  the  ability  to 
evaluate  and  eliminate  large  classes  of  potential  solutions  without 
exhaustive  search  (cf . ,  Amarel,  1980).  A  chess-playing  expert  system,  ft r 
example,  might  consider  whole  sequences  of  moves  or  strategies,  rather 
than  individual  moves,  and  evaluate  the  board  positions  to  which  they  lead 
in  terms  of  relatively  subjective  attributes  like  "domination  of  center 
positions."  More  generally,  expert  systems  make  primary  use  of 
domain- specific  judgmental  knowledge  (i.e.,  heuristics)  rather  than 
reasoning  methods  or  brute -force  search  techniques. 

As  a  result,  in  typical  expert  systems  applications,  the  highest 
available  standard  of  reasoning  in  the  relevant  domain  is  expert  practice 
itself;  and  much  of  the  effort  in  expert  system  development  consists  in 
the  extraction  of  specialized  knowledge  from  human  experts  and  its 
translation  into  machine-usable  form.  Typically,  a  computer  scientist  (or 
knowledge  engineer)  is  required  to  serve  as  mediator  between  the  domain 
expert  and  the  computer  in  the  process  of  "authoring"  a  knowledge  base. 

The  knowledge  engineer  must  identify  and  define  the  problem,  and  develop 
an  initial  understanding  of  the  problem  domain.  Then  he  or  she  must 
tentatively  choose  an  appropriate  knowledge  representation,  reasoning 
mechanism,  and  user-system  interface;  elicit  the  knowledge  from  experts, 
revising  if  necessary  his  or  her  initial  design  choices;  and  validate  the 
system.  Progress  in  automating  the  knowledge  engineer's  role--i.e., 
machine  learning  from  direct  exposure  either  to  the  expert  or  to  the 
domain- -has  been  limited. 
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Why  is  knowledge  elicitation  hard?  Ve  would  argue  that  knowledge 
elicitation  is  seldom  (if  ever)  the  literal  "transfer"  of  information  from 
expert  to  system: 

•  No  matter  how  much  time  is  spent  with  the  expert,  the  data 
will  always  be  incomplete  and  compatible  with  a  variety  of 
different  hypotheses  regarding  his  knowledge  and  reasoning 
methods.  The  knowledge  engineer  must  fill  in  gaps,  identify 
possible  lapses  or  errors,  and  extrapolate  from  the  observed 
set  of  problems  to  others  that  may  occur  in  the  domain.  (This 
is  true  even  if  the  knowledge  engineer  is  the  expert.) 

•  Often,  expertise  has  been  developed  over  a  unique  and  lengthy 
series  of  problem-solving  experiences  in  a  domain.  Such 
knowledge  does  not  permit  easy  verbal  articulation,  even  by 
the  expert. 

•  Some  of  the  information,  on  the  other  hand,  may  be  in 
textbooks  or  other  articulable  form.  Indeed,  experts  often 
have  command  over  a  multiplicity  of  representations,  knowledge 
sources,  and  problem-solving  strategies.  These  must  be  sorted 
out,  their  proper  roles  identified,  and  apparent 
inconsistencies  resolved. 

•  The  knowledge  engineer  must  work  with  elicitation  methods 
which  may  themselves  distort  or  influence  the  character  of  the 
expert's  inputs  in  unknown  ways. 

•  The  knowledge  engineer  must  translate  the  expert's  knowledge 
into  computer  implementation  formats  that  are  limited  in  their 
ability  to  capture  the  flexibility  and  variety  of  human 
concepts  and  reasoning  methods . 

•  Expert  knowledge  is  typically  "heuristic”  or  judgmental. 
Methods  for  capturing  expert  judgment  may  yield  results  that 
apparently  diverge  from  normatively  justified  principles  for 
reasoning  (e.g.,  the  work  of  Kahneman  and  Tversky) .  What  is 
at  fault:  the  experts,  the  normative  principles,  or  the 
elicitation  methods? 

•  Multiple  experts  in  the  same  domain  may  disagree  in  their 
methods,  in  their  conclusions,  or  in  both.  What  criteria  can 
be  applied  for  calibrating  degrees  of  expertness?  When  should 
competing  views  be  reconciled,  and  when  should  the  possibility 
of  alternative  approaches,  and  even  multiple  possible 
conclusions,  be  allowed  for? 

Clearly,  the  metaphor  of  knowledge  elicitation  as  "mining"  for 
"nuggets"  of  knowledge  (e.g.,  Hayes-Roth  et  al.,  1983)  is  misleading. 

There  may  be  no  static  "nuggets"  waiting  to  be  found.  In  our  view, 
elicitation  is  a  process  of  definition  and  redefinition,  in  which  the 
knowledge  engineer  creatively  models  the  observed  data  while  striking  a 
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balance  among  competing  representational  constraints.  In  such  a  process, 
both  the  expert's  and  the  knowledge  engineer's  picture  of  the  knowledge 
may  shift  as  multiple  viewpoints  are  explored. 

2.2  What  Is  Expertise? 

Is  there  something  qualitatively  distinctive  about  the 
representations  and/or  methods  of  reasoning  of  experts  as  compared  to 
non-experts?  Does  the  character  of  knowledge,  as  well  as  its  quantity, 
change  as  a  novice  becomes  an  expert?  It  is  commonplace  that  expertise 
accrues  over  extended  experience  in  a  domain.  It  is  not  so  obvious, 
however,  how  it  accrues  (Is  experience  necessary  but  not  sufficient?  What 
kind  or  kinds  of  experience?  Is  a  certain  type  of  preparedness  also 
required?).  Nor  is  it  clear  what  it  is  that  accrues  or,  in  a  practical 
vein,  how  experts  may  be  identified  for  such  purposes  as  job  assignment, 
expert  system  knowledge  elicitation,  or  codification  of  doctrine. 

Finally,  it  is  not  clear  that  expertise  means  the  same  thing  or  develops 
in  the  same  way  in  different  fields . 

In  rare  instances,  it  may  be  possible  to  collect  sufficient 
quantities  of  representative  data  under  sufficiently  controlled 
circumstances  to  distinguish  experts  from  non-experts,  by  significantly 
superior  performance  measured  against  a  relatively  objective  standard 
(e.g.,  number  of  surviving  patients  with  a  specific  ailment).  More  often, 
however,  such  data  are  unavailable  either  in  practice  or  in  principle  (for 
example,  expertise  in  military  tactical  intelligence  might  only  be  fully 
tapped  under  warfare  conditions,  but  an  objective  ground- truth  measure 
would  only  be  available  in  peace-time  exercises).  In  these  cases,  the 
only  present  recourse  is  to  utilize  imperfect  indicators:  either 
presumptive  causes  of  expertise  (type  of  training,  years  of  experience)  or 
presumptive  effects  (organizational  rank,  the  consensus  of  colleagues) .  A 
more  theoretically  well-founded  criterion  of  expertise,  in  terms  of 
internal  properties  of  reasoning  or  knowledge,  which  may  appear  in 
virtually  any  sample  of  performance,  might  well  have  important  practical 
implications . 

A  body  of  research  suggests  that  the  development  of  expertise 
follows  two  distinct,  and  apparently  contrary,  paths:  on  the  one  hand, 
toward  an  Increasing  repertoire  of  direct,  prepackaged  responses  to  the 
environment;  and  on  the  other  hand,  toward  a  growing  ability  to  reason 
abstractly  and  flexibly  about  the  problem  domain.  On  the  first  path, 
experts  are  adaptively  controlled  by  the  immediate  environment;  they 
quickly  and  intuitively  "see"  the  right  answers  without  the  need  for  much, 
If  any,  thought.  On  the  second  path,  experts  are  freed  from  the  immediate 
environment  to  plan  further  ahead,  and  to  anticipate  less  directly  related 
consequences.  The  first  path  is  associated  with  bottom-up,  generative 
problem-solving,  in  which  data  directly  trigger  cognitive  processes  that 
lead  to  increasingly  higher-level  conclusions.  The  second  path  is 
associated  with  top-down,  interpretative  problem-solving,  which  utilizes 
models  and  beliefs  at  a  high  level  and  works  downward  to  predictions  about 
the  data.  A  hypothesis  which  we  shall  explore  is  that  generative 
knowledge  is  associated  with  procedural  representations,  which  specify 
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cognitive  operations  in  an  inflexible,  but  highly  efficient  way;  and  that 
interpretative  knowledge  is  associated  with  declarative  representations, 
which  lend  themselves  flexibly,  but  less  efficiently  to  a  variety  of  uses. 
It  is  illuminating  to  consider  how  each  dimension  of  expertise  may  be 
utilized  to  grapple  with  novelty.  In  effect,  there  are  three  levels  at 
which  adaptation  might  take  place: 

•  The  selection  and  shaping  of  (generative)  behavior  by  direct 
experience  of  success  and  failure  in  the  problem  environment ; 

•  The  selection  and  shaping  of  plans  or  hypotheses  by  vicarious 
experience  of  success  and  failure  in  an  interpretative  model 
of  the  problem  environment; 

•  The  exploratory  generation  of  new  interpretative  models  and 
their  evaluation  in  terms  of  the  accuracy  of  the  predictions 
to  which  they  lead. 

These  three  forms  of  learning  represent  increasing  levels  of 
adaptiveness  to  potential  changes  in  the  environment,  and  decreasing 
levels  of  efficiency  in  dealing  with  routine  or  familiar  aspects  of  the 
environment.  Their  relative  prominence  in  any  particular  application  may 
well  depend  on  the  nature  of  the  problem  domain,  the  familiarity  of  the 
problem,  and  perhaps  on  the  cognitive  style  of  the  expert. 

In  the  remainder  of  Section  2.0,  we  briefly  consider  each  type  of 
expertise  and  then  turn  to  some  implications  for  the  elicitation  process. 

2.2.1  Generative  expertise.  Expertise  is  often  equated  with 
"know-how,"  and  one  difference  between  experts  and  novices  may  lie  in  the 
distinction  between  knowing  how  and  knowing  that.  Extensive  practice 
(e.g.,  riding  a  bicycle,  speaking  a  foreign  language)  may  lead  to  a 
direct,  "intuitive"  skill  on  the  part  of  the  expert,  which  he  himself  is 
unable  to  articulate  or  explain.  While  the  novice  may  be  able  to  express 
many  facts  about  the  domain  (i.e.,  know  that...,)  he  may  lack  this 
essential  know-how. 

Two  recent  theories  have  attempted  to  account  for  expertise  along 
these  lines:  the  "compilation"  of  knowing  how  from  knowing  that,  and  the 
ability  to  perceive  the  environment  in  increasingly  large  "chunks." 

Any  symbol  system,  whether  human  or  computer,  can  be  regarded  as 
consisting  of  (a)  a  set  of  data  structures  and  (b)  the  processes  which 
interpret  and  manipulate  them  in  order  to  draw  conclusions  (cf . ,  Anderson, 
1978;  Newell,  1981).  Anderson  (1982)  has  proposed  a  detailed  cognitive 
architecture  in  which:  consciousness  (knowing  that)  is  identified  with 
the  data  in  working  memory;  procedural  knowledge  (knowing  how)  is 
represented  in  the  form  of  if-then  production  rules;  and  the  development 
of  cognitive  skill  involves  a  shift  from  (a)  explicit  declarative 
knowledge  to  (b)  implicit  procedural  knowledge.  The  first  phase  of  skill 
development  is  declarative :  domain-  or  problem- specific  facts  are 
retrieved  from  long-term  memory,  explicitly  represented  in  working  memory, 
and  interpreted  by  general-purpose  problem-solving  procedures.  The 
declarative  information  in  working  memory  triggers  the  application  of 
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"if-then"  production  rules,  or  procedures,  which  in  turn  activate  new 
declarative  information  triggering  new  rules,  and  so  on.  Through 
successful  repetition,  a  process  akin  to  the  compilation  of  a  computer 
program  occurs:  specialized  procedures  are  created  which  take  over  the 
functions  of  the  declarative  knowledge.  One  subprocess  involved  in 
compilation  is  composition,  which  collapses  rules  that  tend  to  be 
triggered  in  sequence  into  a  single  procedure.  Another  subprocess, 
proceduralization,  replaces  generally  stated  rule  conditions  (that  require 
specialized  declarative  knowledge  for  their  use)  with  specific  rule 
conditions . 

Figures  2-1  through  2-3  illustrate  very  roughly  the  successive 
stages  of  expertise  according  to  Anderson's  theory.  Notice  that  for  the 
novice  (Figure  2-1),  virtually  all  the  knowledge  pertaining  to  attack 
planning  must  be  retrieved  from  long-term  memory  and  explicitly 
represented  in  working  memory.  By  contrast,  in  the  final  stage  of 
development  (Figure  2-3)  this  same  domain  knowledge  is  directly  embodied 
in  processes.  (According  to  Anderson,  procedural  knowledge  may  continue 
to  improve  through  processes  of  discrimination  and  generalization.) 
Anderson  links  his  concept  of  procedural  knowledge  with  the  notion  of 
" automat icity"  in  Schneider  and  Shiffrin  (1977).  Automatic/procedural 
performance  is  faster  (since  it  requires  fewer  steps)  and  less  likely  to 
interfere  with  the  parallel  performance  of  other  tasks  (since  it  demands 
less  space  in  working  memory) .  By  the  same  token,  it  is  less  accessible 
to  conscious  awareness. 

Experimental  data  suggest  that  expert  knowledge  may  in  fact  undergo 
a  developmental  sequence  of  this  kind.  Most  of  these  data  come  from 
think-aloud  problem  solving  protocols.  For  example,  in  a  study  comparing 
expert  and  novice  performance  on  physics  problems,  Larkin  et  al.  (1980) 
found  that  novices  used  general  problem-solving  methods,  e.g.,  they  worked 
backward  from  the  unknown  through  various  subgoals  (e.g.,  solve  an 
equation  containing  the  unknown)  to  the  given  quantities.  Novices 
explicitly  mentioned  the  equations  used  at  each  stage.  Experts,  by 
contrast,  were  faster,  worked  forward  from  the  given  to  the  desired 
quantities,  and  usually  verbalized  only  numerical  results  rather  than  the 
equations  themselves.  These  findings  suggest  that  novices  have 
domain-specific  declarative  knowledge  (viz.,  knowledge  Chat  certain 
equations  are  useful  for  various  purposes)  and  general-purpose  procedural 
knowledge  for  applying  it  ( e.g .,  knowing  how  to  do  means-ends  analysis); 
but  that  experts  store  entire  domain- specific  solution  methods  as  unitary 
procedures.  In  sum,  experts  substitute  relatively  shallow  chains  of 
automatic  stimulus -response  relationships  for  the  more  elaborate  and 
self-conscious  problem  solving  of  sophisticated  novices. 
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•  Goal  ■  plan  an  attack  with  1st  and  2nd  Brigades 

•  3-to-1  ratio  necessary  for  successful  sttack 


If  A  is  a  goal  and  doctrine  says 
B  is  required  for  A,  adopt  B  as 
subgoal. 


•  Subgoal  *  Achieve  3-to-1  force  ratio  for  1st  and  2nd  Brigades 

•  Current  ratio  for  1st  Brigade  ■  3-to-1, 

for  2nd  Brigade  «  2-to-1, 

(fro*  status  reports) 


If  A  is  goal  and  B  is  current 
situation,  find  difference  C 
between  A  and  B  and  make  reducing 
C  a  siiigoal. 


•  Siiigoal  *  reducing  difference  between  3-to-1  and  2-to-1  ratios 

for  2nd  Brigade 

•  Reinforcement  is  a  method  for  increasing  force  ratio 

(from  doctrine) 

•  Reinforcement  consists  of  moving  troops  from  one  unit  to 
another  unit  that  is  to  be  reinforced  (semantic  knowledge) 


If  a  goal  involves  n  objects  and 
some  of  the  objects  are  known, 
make  finding  the  unknown  objects 
a  subgoal. 


SUbgoal  *  find  a  unit  to  serve  as  a  source  of 
reinforcements  for  2nd  Brigade. 

Infantry  units  should  be  reinforced  with  tanks. 
Tank  wits  should  be  reinforced  with  infantry. 

(from  doctrine) 


If  A  is  a  goal  and  doctrine  says 
B  is  required  to  achieve  A  if  C 
is  true,  and  D  is  required  to 
achieve  A  if  E  is  true,  then  find 
out  if  C  or  E  is  true. 


2nd  Brigade  is  infantry  (from  long-term  memory) 


If  A  is  a  goal  and  doctrine  says 
B  is  required  for  A,  then  make  B 
a  subgoal. 


Subgoal:  find  a  tank  wit  as  source  of  reinforcements 

1-49  Armored  Battalion  is  a  tank  wit  (from  long-term 
memory) 


If  A  is  a  goal  and  a  way  to 
achieve  A  has  been  found,  then 
consider  the  goal  success. 


(All  goals  marked  as  successful) 

ACTION  ■  Reinforce  2nd  Brigade  wit  with  1-49  Armored 
Battalion 


Figure  2-1.  A  Novice’s  Declarative  Knowledge  Representation. 
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Goal  -  plan  an  attack  with  1st  and 
2nd  Brigades 

Current  ratio  of  2nd  Brigade  -  2-to-l 
(from  reports) 

2nd  Brigade  is  an  infantry  unit 
1-49  Armored  Battalion  is  a  tank  unit 


If  the  goal  is  to  attack  with  an 
infantry  unit  and  the  ratio  of 
troops  is  less  than  3-to-l,  then 
reinforce  with  a  tank  unit. 


Reinforce  2nd  Brigade  with  1-49  Armored 
Battalion 


Figure  2-2.  After  Composition. 


(Explicit)  Declarative  Knowledge  (Implicit)  Procedural  Knowledge 


Goal  -  plan  an  attack  with  1st  and 
2nd  Brigades 

Current  ratio  of  2nd  Brigade  -  2-to-l 
(from  reports) 

If  the  goal  is  to  attack  with 
2nd  Brigade  and  the  ratio  of 
troops  is  less  than  3-to-l,  then 
reinforce  with  1*49  Armored 
Battalion. 


Reinforce  2nd  Brigade  with  1-49 
Armored  Battalion 


Figure  2-3.  After  Proceduralization. 
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Other  research  suggests  that  implicit  procedural  knowledge  may 
evolve  directly  £rom  experience  in  a  domain,  without  an  initial  stage  of 
explicit  declarative  representation.  Berry  and  Broadbent  (1984)  found 
that  practice  in  a  complex,  computer- implemented  control  task  improved 
performance,  but  did  not  improve  subjects'  ability  to  answer  questions 
about  the  task.  Verbal  instructions  had  the  converse  effect.  Moreover, 
there  was  a  negative  correlation  between  subjects'  ability  on  the  task  and 
their  ability  to  answer  explicit  questions  about  it,  when  task  solutions 
were  not  obvious  or  salient  If  task  knowledge  were  originally  represented 
in  declarative  form,  a  positive  correlation  would  be  expected- -assuming 
that  the  original  declarative  knowledge  could  be  reliably  recalled  and 
verbalized. 

It  may  not  be  safe  to  assume,  however,  that  declarative  knowledge  is 
always  accurately  describable.  Declarative  knowledge  includes  not  only 
generalizations  (such  as,  a  3-to-l  force  ratio  is  necessary  for  a 
successful  attack) ,  but  also  internal  representations  of  particular 
successful  or  unsuccessful  problem-solving  episodes .  Such  episodes  may 
involve  examples  of  problem-solving  by  others,  or  one's  own 
trial -and- error  experiences.  There  may  be  general-purpose  procedures  for 
utilizing  such  episodic  memories  for  the  selection  of  a  response  in  new, 
similar  problems  (Anderson,  Greeno,  Kline,  and  Neves,  1981).  Attempts  to 
describe  such  episodes,  however,  may  invoke  different  procedures,  which 
depend  more  heavily  on  conventional  or  stereotypical  notions  of  how  the 
task  should  be  solved,  and  therefore  focus  on  different  aspects  of  the 
remembered  situation.  There  is  considerable  evidence  regarding  the 
unreliability  of  "autobiographical"  episodic  memory  (e.g.,  Loftus, 
Fienberg,  and  Tanur,  1985;  Bradburn,  Rips,  and  Shevell,  1987). 

A  second  theory  of  expertise,  which  also  associates  expert  knowledge 
with  increased  automaticity ,  involves  the  notion  of  chunking.  Introduced 
by  Miller  (1956) ,  chunking  is  the  ability  to  organize  a  number  of  items  of 
information  into  a  larger  grouping,  which  is  then  treated  as  a  single 
unit.  DeGroot  (1965)  utilized  this  idea  to  explain  the  knowledge  of  chess 
masters.  DeGroot  found  that  masters  far  outperform  novices  in  the 
short-term  recall  of  chessboard  patterns.  This  advantage  disappeared, 
however,  when  the  chessboard  configurations  to  be  recalled  were 
constructed  randomly,  suggesting  that  chess  masters  are  not  simply  better 
at  memorizing  the  location  of  individual  pieces.  Rather,  they  are  able  to 
recognize  and  identify  familiar  structural  relations  in  the  legal 
chessboard  patterns.  DeGroot  concluded  that  expertise  in  chess  lies  in 
direct,  automatic  linkages  between  these  perceived  structural  relations 
and  learned  strategies  or  responses.  Fast  unconscious  processing  thus 
replaces  the  novice's  conscious,  slow  reasoning. 

Newell  and  Rosenbloom  (1981)  have  proposed  a  general  theory  of  skill 
(perceptual,  motor,  and  cognitive)  based  on  chunking.  In  a  wide  variety 
of  task  domains,  including  problem-solving,  it  has  been  found  that  the 
time  required  to  perform  a  task  is  a  power  function  of  the  number  of 
trials  of  practice  at  tasks  of  that  type.  Newell  and  Rosenbloom  explain 
the  improvement  due  to  practice  in  terms  of  a  constant  rate  at  which  task 
elements  are  encoded  into  larger  chunks.  A  power  function  (rather  than 
the  expected  exponential  function)  can  be  derived  on  the  hypothesis  that 
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chunks  formed  later  In  practice  are  larger  than  those  formed  earlier; 
hence,  they  are  more  specialized,  i.e.,  less  likely  to  recur  and  be  used 
in  the  solution  of  subsequent  problems, 

2.2.2  Declarative  expertise.  Compilation  and  chunking  have  their 
primary  effect  on  the  speed  of  performance,  by  collapsing  large  numbers  of 
operations  into  single  routines.  Qualitative  changes  in  performance 
(e.g.,  working  forward  to  the  solution  rather  than  working  backward  from 
the  goal)  are  incidental  by-products.  Improvements  in  the  level  of 
performance  are  also  by-products,  resulting  from  the  ability  to  generate  a 
larger  number  of  possible  solutions  in  a  shorter  period  of  time. 

Moreover,  there  is  a  self-limiting  character  to  the  improvements  created 
by  compilation  and  chunking:  the  increasingly  specialized  structures  they 
produce  have  a  correspondingly  decreased  range  of  useful  applications . 

Nevertheless,  there  is  evidence  that  the  superiority  of  experts  goes 
beyond  the  rapidity  with  which  operations  are  performed.  Experts  appear 
to  have  knowledge  structures  that  are  both  more  general  and  more  flexible 
than  those  of  novices.  Evidence  for  the  special  character  of  expert 
representations  comes  from  a  variety  of  studies,  utilizing  similarity 
judgments,  clustering  in  recall,  and  think- aloud  problem  solving. 

Similarity  judgment.  In  physics,  Chi  et  al.  (1981)  found  that 
experts  and  novices  differ  in  the  way  they  sort  problems  by  similarity. 
Novices  categorize  problems  by  "surface  structure,"  i.e.,  superficial 
features  such  as  type  of  apparatus,  while  experts  rely  on  basic  principles 
of  physics  (e.g.,  conservation  of  energy)  and  generic  solution  techniques 
associated  with  such  principles.  Similar  differences  between  experts  and 
novices  in  algebra  are  reported  by  Shoenfeld  and  Herrmann  (1982):  algebra 
experts  sort  problems  by  solution  method,  while  novices  depend  on  words  or 
objects  mentioned  in  the  problem  statement.  Weiser  and  Shertz  (1983) 
replicated  these  findings  for  computer  programming:  experts  sorted 
programming  problems  by  reference  to  the  types  of  algorithms  that  would  be 
employed  in  solving  them,  while  novices  sorted  by  reference  to  the 
application  area  from  which  the  problem  was  drawn. 

Clustering  in  recall.  Chase  and  Simon  (1973)  analyzed  pauses  in  the 
recall  of  chessboard  configurations,  as  well  as  visual  glances  in  the 
perceptual  reconstruction  of  chessboard  configurations.  The  clusters 
identified  in  this  way  for  chess  masters  suggested  that  the  meaningful 
patterns  within  the  chessboard  were  classified  and  understood  at  an 
abstract  level,  e.g.,  as  "attack"  configuration  or  "defense" 
configuration.  McKeithen  et  al.  (1981)  obtained  similar  results  in  a 
study  of  clustering  in  the  recall  of  computer  programming  concepts  by 
expert  and  novice  programmers.  Adelson  (1984)  replicated  the  results  in  a 
study  of  the  order  in  which  scrambled  lines  of  a  computer  program  were 
recalled:  novices  clustered  by  syntactic  similarity,  while  experts 
grouped  lines  that  formed  semantically  meaningful  program  modules. 

Think-aloud  problem-solving .  Evidence  that  abstract  concepts  are 
actually  used  in  expert  problem  solving  comes  from  think-aloud  protocols. 
In  physics  (Larkin,  1981)  and  geometry  (Greeno,  1983),  the  entities 
mentioned  in  expert  as  compared  with  novice  protocols  tend  to  be  technical 
rather  than  familiar,  and  to  be  closely  tied  to  fundamental  laws.  Larkin 
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et  al.  (1980)  and  Novack  and  Araya  (1981)  found  that  physics  experts 
associate  specific  types  of  problems  with  idealized  spatial 
representations  of  the  objects  and  relations  involved  in  those 
problem- types.  In  solving  a  problem,  experts  utilize  qualitative  rules  to 
infer  unknown  properties  within  these  representations  from  those  that  are 
known,  rather  than  working  explicitly  with  equations.  Larkin  (1981) 
identifies  a  number  of  aspects  in  which  expert  inferential  processes  are 
more  general,  and  therefore  more  powerful  than  those  of  novices:  e.g.,  (a) 
properties  of  objects,  as  perceived  by  experts,  are  local  rather  than 
defined  in  terms  of  expected  effects  on  other  objects,  thus  permitting  use 
of  knowledge  about  these  objects  across  a  wider  range  of  contexts;  (b) 
inference  by  experts  involves  atemporal  constraints  as  opposed  to 
single-direction  temporal  simulation,  thus  permitting  the  same  parameter 
to  serve  as  an  input  in  one  problem  context  and  as  an  output  in  another; 
(c)  the  same  solution  can  be  arrived  at  in  more  than  one  way,  utilizing 
different  inputs . 

These  and  other  findings  suggest  that  experts  not  only  respond 
directly  and  intuitively  to  perceived  structure,  but  actively  utilize 
cognitive  structures  to  organize  their  knowledge  and  to  derive  predictions 
and  expectations.  Improved  capacity  for  top-down  processing  thus  seems  to 
characterize  expertise  as  much  or  more  than  improved  bottom-up  processing. 

An  important  additional  feature  of  this  top-down  processing, 
particularly  emphasized  by  Larkin,  is  the  flexibility  or  redundancy  that 
it  incorporates.  It  is  this  feature  that  suggests  an  advantage  of 
declarative  over  procedural  representations  for  such  knowledge.  The  key 
distinction  between  the  two  is  that  procedural  representations  specify  the 
use  to  which  the  knowledge  is  to  be  put;  thus,  each  potential  use  requires 
a  separate  representation.  Declarative  representations,  by  contrast,  can 
be  utilized  (by  general-purpose  procedures)  in  a  variety  of  ways. 

For  example,  one  type  of  declarative  representation,  the  script 
(Schank  and  Abelson,  1977) ,  represents  a  stereotypical  sequence  of  events 
associated  with  some  activity  or  goal.  An  extremely  simplified  example 
might  be  the  following  attack  script: 


Attack 


artillery  troops  attack  movement 

forward  across 

forward  line 


This  same  script  can  be  utilized  either  for  understanding  a 
situation  or  for  planning  one's  own  actions,  while  large  numbers  of 
different  production  rules  would  be  required  for  each  purpose.  In 
situation  understanding,  if  any  one  of  the  specified  events  is  observed, 
the  others  can  be  inferred  (with  some  degree  of  uncertainty):  e.g.,  after 
a  massing  of  enemy  troops  is  observed,  we  may  infer  that  artillery  have 
been  already  moved  forward,  and  we  may  expect  an  artillery  attack  and 
forward  troop  movements  in  the  near  future.  Similarly,  if  an  artillery 


attack  is  observed,  we  expect  troop  massing  to  have  already  taken  place, 
and  troop  movements  to  be  imminent.  By  contrast,  in  a  production  rule 
representation,  separate  rules  would  be  required  for  each  possible  input. 
In  planning  an  attack,  similarly,  the  same  script  may  be  utilized  to 
identify  and  organize  actions  that  we  must  take  at  any  stage  of  the 
attack.  Finally,  note  that  the  same  knowledge  structure  could  be  used  (in 
conjunction  with  linguistic  procedures)  to  generate  verbal  descriptions  of 
an  attack  or  to  answer  questions. 

It  is  clear  that  procedural  and  declarative  representations  reflect 
contrasting  criteria  of  efficiency.  Procedural  representations  maximize 
the  speed  of  a  specific,  adaptive  set  of  responses  to  a  specific  set  of 
stimuli.  Declarative  representations ,  on  the  other  hand,  maximize  the 
amount  of  information  that  is  communicated  in  a  given  period  of  time.  Any 
part  of  a  declarative  representation  can  lead  (perhaps  with  some 
uncertainty)  to  a  reconstruction  of  the  whole  (Bruner,  1956,  p.46). 
Declarative  structures,  then,  capture  the  redundancies  in  the  problem 
solver's  environment. 

Eleanor  Rosch  (1983,  1978;  Rosch  et  al.,  1976)  has  provided 
empirical  support  for  a  view  of  this  sort,  with  respect  to  categories  in 
natural  language.  According  to  Rosch,  objects  are  not  simply  arbitrary 
ways  of  slicing  up  the  world  into  classes.  In  such  taxonomies  there  is 
generally  one  level  of  abstraction  that  is  more  basic  than  the  others, 
i.e.,  learned  earlier,  and  used  most  often  in  acting  upon,  thinking  about, 
or  naming  the  object.  For  example,  terms  like  dog,  cat,  and  lion,  are  at 
the  basic  level  of  abstraction,  by  contrast  with  more  abstract  terms  like 
mammal  or  animal  and  more  concrete  terms  like  spaniel  or  collie.  Rosch 
proposes  an  explanation  of  this  phenomenon  in  terms  of  the  adaptiveness  of 
categories  to  the  correlational  structure  of  cues  in  the  environment.  The 
basic  level  of  abstraction  is  the  most  inclusive  level  in  the  taxonomy  at 
which  there  are  attributes  common  to  most  or  all  of  the  members  of  the 
category.  At  too  high  a  level  of  abstraction  (e.g.,  mammal),  there  is 
less  similarity  among  members  of  the  category,  and  as  a  result  we  are  less 
able  to  predict  what  the  object  is  like  (or  what  actions  we  should  take 
with  regard  to  it)  from  the  category  name .  At  too  low  a  level  of 
abstraction  (e.g.,  collie),  there  are  too  many  properties  and  actions)  in 
common  with  other  categories  (e.g.,  other  types  of  dogs),  and  cognitive 
economy  is  unnecessarily  sacrificed.  Basic- level  categories  thus  best 
achieve  the  two  purposes  of  a  categorization  system:  (1)  to  enable  us  to 
predict  as  many  properties  as  possible  from  any  other,  and  (2)  to  bring 
pragmatically  useful  order  to  the  Infinite  differences  among  stimuli. 

Basic  categories  provide  the  most  information  with  the  least  effort. 

It  seems  plausible  to  suppose  that  the  development  of  expertise  is, 
in  part  at  least,  a  continuation  and  intensification  of  the  process 
involved  in  language  learning.  The  categories  employed  by  the  expert, 
while  more  abstract  and  technical,  give  him  a  more  powerful  tool  for 
exploiting  regularities  in  his  environment. 
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3.0  KNOWLEDGE  STRUCTURES  AND  ELICITATION  STRATEGIES 


Psychological  studies  of  expert  problem-solving  behavior  give 
evidence  of  an  interactive  use  of  both  high-level,  relatively  abstract 
domain  knowledge  and  goal-directed  planning  with  lower-level,  more 
concrete  and  detailed  information  and  rules  of  action.  It  would  follow 
that  methods  of  elicitation  geared  to  capture  knowledge  at  those  different 
levels  of  abstraction  would  be  more  effective  in  the  acquisition  of  the 
problem-solving  knowledge  that  real  experts  use,  than  standard  elicitation 
approaches  that  do  not  recognize  explicitly  the  use  of  diverse  forms  of 
knowledge . 

In  this  section,  interpretative  and  generative  types  of  knowledge 
will  be  described  at  greater  length,  along  with  a  description  of  various 
knowledge  structures  amenable  to  the  representation  of  interpretative 
versus  generative  knowledge,  and  how  these  structures  can  be  integrated. 
This  will  be  followed  by  a  general  discussion  of  elicitation  strategies 
and  candidate  strategies  that  promise  to  be  effective  in  eliciting  these 
two  different  kinds  of  knowledge.  We  will  then  consider  the  interaction 
of  interpretative  (top-down)  and  generative  (bottom-up)  knowledge  in  the 
course  of  problem  solving,  in  conjunction  with  methods  for  elicitation  of 
problem  control  knowledge  and  appropriate  structures  for  the 
representation  of  the  flow  of  control  and  attention  across  levels  of 
abstraction  in  the  process  of  solving  domain  problems.  Finally,  we 
discuss  some  other  issues  that  are  important  to  the  ultimate  success  of 
elicitation  concerning  the  purpose  of  the  expert  knowledge  elicitation  and 
the  nature  of  "expertise." 

3 . 1  Interpretative  Versus  Generative  Knowledge 

Interpretative  knowledge  involves  the  high-level  goals,  plans,  and 
strategies  needed  for  problem  solving  within  a  domain,  to  provide  an 
overall  direction  for  the  more  detailed  problem  solving  required  to 
achieve  intermediate  subgoals.  Interpretative  knowledge  also  refers  to 
the  complex,  semantically  rich  units  of  knowledge  about  objects  and 
processes,  on  which  people  base  actions,  assumptions,  and  expectations. 

For  example,  most  Army  officers  have  some  interpretative  knowledge  about 
how  to  conduct  rear  area  protection  (RAP)  operations,  and  if  an  ammunition 
supply  unit  receives  a  report  of  Level  II  enemy  activity  behind  friendly 
lines,  the  commander  can  assume  his  unit  is  a  potential  target.  He  would 
expect  the  threat  to  consist  of  small  teams  of  special  forces,  using  small 
automatic  weapons  and  explosives,  without  armored  vehicle  support.  They 
would  probably  attack  in  darkness  to  achieve  surprise,  attempt  to  destroy 
as  much  as  possible,  and  then  withdraw  within  half  an  hour.  Upon 
receiving  the  report  of  enemy  activity,  the  commander  would  plan  to 
increase  perimeter  guards,  rehearse  the  unit  quick  response  team,  and 
notify  the  rear  area  operations  center  (RAOC) .  He  would  seek  to  secure 
his  area,  prevent  or  minimize  enemy  interference  with  his  mission,  and 
help  the  RAOC  to  find,  fix  and  destroy  the  enemy  threat.  These 
expectations,  plans  and  goals  will  serve  to  direct  the  more  specific  and 
concrete  actions  that  the  commander  will  take  to  perform  rear  area 
protection. 
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Problem-solving  that  involves  the  use  of  this  sort  of  contextual  or 
general  knowledge  has  also  been  referred  to  as  top-down  processing  because 
the  high-level  general  knowledge  largely  determines  the  interpretation  of 
(and  thus  the  response  to)  low-level  perceptual  units. 

Generative  knowledge  is  more  concrete,  specific  and  perceptual. 
Objects  and  processes  at  this  low  level  are  components  from  which  patterns 
are  generated,  and  subsequently  recognized  and  acted  upon.  Referring  to 
the  rear  area  protection  example  again,  if  a  guard  post  alerted  the  base 
defense  operations  center  (BDOC)  of  suspected  movement,  the  commander 
would  probably  send  a  patrol  to  investigate.  If  the  unit  came  under  small 
arms  fire,  the  commander  would  order  all  soldiers  to  their  fighting 
positions  and  alert  the  RAOC.  If  the  unit  began  receiving  indirect  or 
heavy  weapons  fire,  he  would  request  support  from  the  RAOC,  or  send  his 
quick  response  team  to  destroy  the  source  of  the  firing.  If  the  unit 
perimeter  were  penetrated,  he  might  initiate  emergency  action  procedures 
such  as  destruction  of  nuclear  warheads,  or  call  for  artillery  or  air 
strikes  on  the  overrun  positions.  Thus,  generative  knowledge  is  concerned 
with  the  piecing- together  of  specific  and  concrete  information  in  order  to 
make  an  inference  or  to  determine  a  course  of  action,  also  referred  to  as 
bottom-up  processing. 

3 . 2  Structures  for  the  Representation  of  Interpretative  and  Generative 
Knowledge 

In  this  section,  we  will  describe  a  variety  of  knowledge  structures 
with  which  expert  knowledge  can  be  represented  once  it  has  been  elicited. 
Although  there  are  several  candidate  structures  for  interpretative 
knowledge  and  for  generative  knowledge,  each  individual  structure  has  some 
shortcomings  as  well  as  strengths  in  its  ability  to  represent  expert 
knowledge  at  a  given  level  of  abstraction.  Thus,  our  discussion  will  also 
consider  strategies  for  the  amalgamation  of  different  top-down  versus 
bottom-up  structures,  to  produce  integrated  approaches  to  the 
representation  of  interpretative  versus  generative  knowledge. 

3.2.1  Interpretative  knowledge  structures.  The  structures  that 
will  be  considered  for  the  purpose  of  capturing  top-down  problem-solving 
knowledge  include  scripts,  frames,  and  high-level  mental  models. 

Although  it  has  sometimes  been  argued  that  each  of  these  structures 
could  individually  accommodate  the  range  of  knowledge  needed  for 
intelligent  problem  solving,  they  are  very  well  suited  to  address  the 
questions  of  "How?"  "What?"  and  "Why?"  respectively.  Scripts  provide 
descriptions  of  sequences  of  actions  to  obtain  goals,  frames  provide  rich 
and  detailed  descriptions  of  objects,  and  interpretative  mental  models 
provide  high-level  descriptions  of  causal  processes  involving  a  particular 
set  of  objects  and  prompting  particular  actions. 

Scripts.  People  appear  to  make  use  of  scripts  (Schank  and  Abelson, 
1977)  when  planning  actions  for  problem  solving  or  forming  expectations 
about  future  sequences  of  events.  A  script  has  been  defined  as  a  "causal 
chain"  that  describes  appropriate  sequences  of  events  in  a  particular 
context.  It  is  made  up  of  slots  and  requirements  about  what  can  fill 
those  slots.  What  is  in  one  slot  affects  what  can  be  in  another. 
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Figure  3-1  gives  an  example  of  a  script  representation  that  provides 
a  rather  simplified  description  of  the  G2's  procedure  for  requesting 
aircraft  reconnaissance.  The  script  slot  contains  a  description  of  the 
general  area  of  operations  to  which  the  process  description  pertains.  The 
track  slot  identifies  the  specific  task  of  interest.  Roles  should  contain 
a  listing  of  all  actors  who  will  play  a  part  in  the  sequence  of  events 
that  comprise  the  task.  Props  contain  a  list  of  all  the  objects  that  will 
play  a  part  in  the  sequence  of  actions  described  in  the  script.  Entry 
conditions  describe  the  features  of  the  problem  or  situation  that  are 
necessary  for  this  particular  script  to  be  applied.  Results  describe  the 
net  outcome  of  the  sequence  of  actions  described  in  the  script.  This  is 
generally  a  description  of  players  or  props  that  undergoes  change  in  the 
course  of  actions  in  the  script. 

The  information  described  in  the  aforementioned  slots  provides  a 
preliminary  introduction  and  summary  of  the  more  detailed  sequence  of 
actions  that  follows.  The  remainder  of  a  script  consists  of  a  series  of 
scenes  (rather  like  the  script  of  a  play,  but  without  much  dialogue)  that 
describes  the  required  sequence  of  subtasks  and  procedures  for  performing 
them. 


While  script  structures  provide  a  very  useful  format  for 
representation  of  familiar  processes  at  various  desired  levels  of  detail, 
they  are  not  readily  changed  (McArthur,  Davis  and  Reynolds,  1987),  nor  do 
they  provide  the  apparatus  for  handling  totally  novel  situations.  They 
are  predetermined,  stereotyped  sequences  of  actions  that  define  well-known 
situations,  but  do  not  explicitly  account  for  causal  processes.  Thus, 
there  are  real  limits  to  their  flexibility.  Scripts  are  therefore  not  a 
particularly  suitable  structure  for  representation  of  processes  driven  by 
a  complex  and  rapidly  changing  environment. 

Object  Frames.  Knowledge  about  an  object  can  be  organized  into  a 
semantically  rich  representation,  referred  to  as  a  frame  (Minsky,  1975)  or 
object  schema  (Anderson,  1980).  The  knowledge  within  a  frame  can  be 
hierarchical  in  structure,  and  the  frames  for  different  objects  can,  in 
turn,  be  represented  in  the  form  of  a  network.  The  object  frame  nodes  are 
defined  by  a  collection  of  attributes  (slots)  and  values  for  those 
attributes  (e.g.,  a  range-admissible  value,  or  a  default  value).  Although 
different  frames  used  in  the  same  system  must  have  the  same  slots  or 
terminals,  the  particular  set  of  attribute  slots  can  vary  according  to  the 
descriptive  needs  of  the  system.  Examples  of  attribute  markers  are 
HAS-PART,  HAS-COLOR,  HAS-SIZE,  HAS-GOAL  and  category-member  keys,  such  as 
A-KIND-OF,  SIMILAR  TO,  IS-A,  PART-OF. 

In  addition  to  object  frames,  one  can  have  action  frames  specifying 
the  action  (e.g.,  MOVE,  GIVE,  TAKE,  etc.)  and  the  set  of  objects  involved, 
such  as  an  ACTOR,  an  OBJECT,  action  SOURCE,  DESTINATION,  and  relevant 
RESULTS  and/or  SUBPROCESSES. 
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SCRIPT:  describes  appropriate  sequence  of  events  in  a  particular 

context. 


Script: 

Track: 


intelligence  collection 
air  reconnaissance  request 


Roles:  G2 

G2's  Commander 
Aircraft  Commander 


Props :  map 

current  intelligence  estimate 
request  form 

communications  link  between  players 
aircraft 


Entry  Conditions: 

G2  identifies  serious 
potential  threat 


Results : 

G 2  has  more  info: 
makes  better 
assessment 


G2  needs  high  quality 
intelligence  to  refine 
threat  assessment 


Air  surveill/recon. 
resources  reduced  in 
short  run 


G2  intelligence  needs  is  urgent 

Commander  can  officially  obtain 
aircraft  reconnaissance 


Scene  1:  G2  identifies  info  need  (entry) 

(G2,  current  intell  or  own  force  and  enemy  forces) 
locate  info,  etc.  needs  in  maps,  space  &  time 

Scene  2:  G2  requests  air  reconnaissance  through  Commander 

.  (procedures  for  making  that  request) 


Scene  N:  G2  receives  info  requested  (exit) 


Figure  3-1.  Example  of  a  Script  Representation:  G2's  Procedure 
for  Requesting  Aircraft  Reconnaissance. 
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Frame  representations  have  a  great  deal  of  "default"  information  and 
thus  produce  expectations  about  objects  and  relationships.  These 
properties  make  the  frame  useful  for  data  interpretation  or  "top-down" 
processing.  Figure  3-2  gives  an  example  of  an  object  frame.  Motorized 
Rifle  Regiment  (MRR)  is  an  instance  of  the  category,  "Soviet-Force- 
Element."  This  frame  is  part  of  a  hierarchy  so  it  is  part  of  another 
Soviet  force  element  and  has  distinct  elements  of  which  it  is  comprised. 
Figure  3-3  shows  the  MRR  frame  in  a  hierarchical  network,  where  it  is 
subordinate  to  the  force  element  "tank  division"  and  has  frames  for  "tank 
battalion"  and  "MR  battalion"  in  a  subordinate  (HAS -PART)  relationship  to 
it. 


SOVIET -FORCE -ELEMENT:  Motorized  Rifle  Regiment 
A-KIND-OF:  Motorized  unit 
SIMILAR-TO:  Tank  regiment 

SIZE:  4-8  Battalions  DEFAULT:  6  Battalions 

PART-OF:  Motorized  rifle  division,  tank  division 

HAS-MISSION:  Conduct  offensive  and  defensive  operations  on  nuclear 
and  non-nuclear  battlefield  as  part  of  tank  or  MR 
Division 

HAS -PART:  Hdqtrs ,  MR  BN,  Tank  BN 

COMBAT -SUPPORT:  Artillery  BN,  Air  Defense  BN,  Antitank  Missile  BN, 
Recon.  BN,  Engineer  BN,  Signal  BN 

SERVICE- SUPPORT:  Medical  CO,  Maintenance  CO,  Chemical  Defense  CO, 
Traffic  Control  Platoon,  Supply  &  Service  Platoon 

Figure  3-2.  Example  of  an  Object  Frame:  Soviet  Motorized  Rifle  Regiment. 


Mental  Models.  Knowledge  of  causal  processes,  described  at  a  fairly 
abstract  and  general  level,  can  be  represented  in  terms  of  mental  models 
(M-M)  (Johnson-Laird,  1983;  DeKleer  and  Brown,  1981).  Although  other 
structures  have  been  proposed  for  representation  of  mental  models ,  the 
structure  we  consider  here  is  defined  as  having:  (1)  contextual 
information;  (2)  a  description  of  the  objects  that  are  essential  to  the 
process  being  described  (note:  if  the  object  is  a  person,  this 
description  would  include  the  person's  goals  relevant  to  the  process);  (3) 
the  "forces"  involved  in  the  process,  or  medium  of  interaction  between 
objects;  (4)  a  description  of  the  interaction  between  objects  (i.e., 
forces  acting  on  objects);  and  (5)  outcomes  of  the  process.  This  last 
part  of  the  model  would  typically  consist  of  descriptions  of  the  impact  of 
the  process  on  one  or  more  of  the  objects.  Figure  3-4  illustrates  this 
format  with  a  mental  model  of  the  general  concept  of  "conflict"  between 
two  parties . 
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FORCE_  ELEMENT:  TANK_DMSION 


Figure  3-3.  Soviet  Army  Frames  as  Part  of  a  Network. 
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Contextual 

Information: 

Objects: 


Force: 

Interaction: 
Outcome : 


Red  Force  and  Blue  Force  come  into  contact 

Red  Force  (goal:  achieve  Red  objective, 
prevent  Blue  objective) 

Blue  Force  (goal:  achieve  Blue  objective, 
prevent  Red  objective) 

Red  Force  strength  (property  of  Red  Force) 

Blue  Force  strength  (property  of  Blue  Force) 

Red  Force  strength  vs.  Blue  Force  strength 

If  Red  Force  strength  >  Blue  Force  strength  by  x  % 

Then  Red  objective  achieved,  Blue  objective  prevented 


If  Blue  Force  strength  >  Red  Force  strength  by  y  % 
Then  Blue  objective  achieved,  Red  objective  prevented 


Figure  3-4.  Mental  Model  of  "Conflict". 


Integrated  Interpretative  Knowledge  Representation.  An  integrated 
interpretative  knowledge  representation  would  utilize  scripts,  frames,  and 
mental  models,  since  each  of  these  approaches  to  knowledge  representation 
has,  despite  some  shortcomings,  its  own  individual  strengths  that  are 
rather  complementary  to  the  characteristics  of  the  other  two  structures. 

The  strength  of  script-based  representations  is  that  they  describe 
sequences  of  events  in  a  goal -related  fashion  and  they  are 
expectancy -driven.  This  quality  is  useful  in  representing  problem  solving 
behavior  based  on  standard  operating  procedures  and  Army  doctrine.  The 
weaknesses  of  script  representations  are  three -fold.  First,  scripts 
provide  a  static  as  well  as  stereotypic  description  of  processes  and  thus 
do  not  capture  the  knowledge  used  to  guide  problem  solving  in  dynamic  or 
novel  situations.  Second,  scripts  frequently  involve  the  use  of  objects 
while  not  directly  containing  information  about  them.  For  example,  a 
script  involving  the  conduct  of  a  battle  may  refer  to  the  positioning  of 
battalions  without  describing  what  a  battalion  is.  A  third  weakness  of 
scripts  is  that  while  they  are  goal -directed  in  nature,  the  causal 
relationship  between  the  actions  followed  and  the  desired  goals  is  not 
explicit.  For  example,  the  battle  script  may  include  a  "supply"  scene, 
although  how  this  helps  fighting  the  battle  is  typically  left  out  of  the 
script.  Frames  and  mental  model  representations  can  help  compensate  for 
these  shortcomings,  however.  Frames  are  a  useful  representation  tool  for 
capturing  information  about  the  objects  (e.g.,  objects  that  are  part  of 


23 


scripts) .  Mental  models  can  provide  a  good  vehicle  for  representation  of 
dynamic-process  abstractions  as  well  as  the  causal  relationships  between 
problem-solving  goals  and  actions. 

An  integrated  interpretative  representation  scheme  can  thus  be 
constructed  using  scripts,  frames,  and  mental  models,  building  on  the 
complementarity  of  these  structures.  The  structures  would  be  linked  by 
pointers . 

Emphasis  on  the  use  of  one  structure  versus  another  will  greatly 
depend  on  the  nature  of  the  problem  solving  in  a  particular  domain.  For 
example,  problem  solving  that  is  more  object-oriented  would  probably 
involve  a  predominant  use  of  frames  while  problem  solving  involving  a 
rather  standard  and  consistent  set  of  procedures  and  problem  situations 
could  be  represented  largely  in  terms  of  scripts.  In  domains  where  the 
observable  features  of  problems  can  vary  greatly,  but  the  set  of  possible 
underlying  causal  processes  is  finite,  the  approach  to  representation 
likely  to  dominate  would  be  mental  models . 

3.2.2  Generative  knowledge  structures.  Structures  that  appear  most 
amenable  to  representation  of  an  expert's  knowledge  for  bottom-up 
processing  are  production  rules,  semantic  nets,  and  highly  constrained 
(i.e.,  very  detailed)  mental  models.  These  structures  respectively  serve 
to  address  the  questions  of  "How?"  "What?"  and  "Why?"  at  a  very  concrete 
and  specific  level. 

Production  Rules.  Rule -based  knowledge  representations  consist  of  a 
situation-recognition  part  and  an  action  or  implications  (for  inference) 
part  (Newell  and  Simon,  1972).  This  is  typically  expressed  as  an 
IF-condition-THEN-action  statement.  When  the  current  problem  situation 
satisfies  or  matches  the  IF  part  of  a  rule,  the  reasoning  or  action 
specified  by  the  THEN  part  of  the  rule  is  applied  or  performed  (Nilsson, 
1980) .  The  condition  portion  of  a  production  rule  can  contain  compound 
events,  connected  by  AND  and  OR.  Figure  3-5  offers  some  examples  of 
production  rules  that  might  be  part  of  the  knowledge  base  of  an  Army 
intelligence  expert. 


[This  space  intentionally  left  blank.] 
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A  Coded  Example : 


(1)  IF  ((friendly-unit-loc)  minus  ( enemy - uni t-loc))  <  - 
( friendly- artillery -  range ) 

AND  (enemy-unit-x-status)  -  (uncommitted) 

OR  (enemy-unit-x-status)  -  (stationary) 

AND  (friendly-enemy-contact)  -  (not  reported) 


THEN  (enemy-unit-x-status)  -  (advance- to-contact) 


A  Written  Example : 

(2)  IF  enemy  unit  X  assumed  to  be  advancing  to  contact 

THEN  identify  intelligence  sources  Y  capable  of  observing 
enemy  unit  X 

AND  Request  intelligence  on  enemy  unit  X  from  all  sources  Y 
Figure  3-5.  Production  Rules  Examples. 


A  sequence  of  production  rules  can  produce  an  inference  chain  that 
can  serve  to  make  inferences,  by  forward- chaining,  or  deductions,  by 
backward- chaining.  Figure  3-6  shows  an  AND/OR  tree  of  production  rules 
and  inference  chains. 

Forward- chaining  involves  inferring  general  implications  from 
specific  information.  For  example,  specific  conditions  such  as  "current 
assessment  of  enemy  status  is  that  the  enemy  is  stationary  at  some 
location  X"  and  "location  X  is  within  friendly  artillery  range"  will  lead 
to  a  more  general  condition- -that  the  enemy  situation  assessment  needs  to 
be  updated  and  perhaps  revised.  Backward- chaining  involves  just  the 
opposite;  the  statement  of  a  general  condition  implies  the  pre-existence 
of  some  more  specific  conditions .  A  recognized  need  to  update  or  revise 
the  enemy  situation  assessment  indicates  that  the  current  one  is  perhaps 
obsolete,  which  in  turn  may  imply  that  the  time  lapse  from  the  previous 
assessment  update  is  more  than  1  hour;  it  implies  that  the  current 
assessment  appears  implausible  in  the  light  of  some  new  information,  such 
as  the  main  attack  has  been  projected  along  corridor  A  but  the  ratio  of 
enemy- to -friendly  forces  along  corridor  A  is  much  less  than  that  required 
by  Soviet  doctrine  for  a  successful  attack. 

Semantic  Nets.  Repeated  observation  of  objects  and  events  with 
similar  features  leads  to  the  formation  of  categories,  classes,  and 
types--of  objects,  events,  and  people.  One  representation  of  this  sort  of 
knowledge  is  a  semantic  net.  Semantic  nets,  originally  developed  as 
psychological  models  of  human  memory  (Quillan,  1966;  Rumelhart,  Lindsay 
and  Norman,  1972)  consist  of  points,  called  nodes,  connected  by  links, 
called  arcs,  which  describe  the  relationships  between  nodes.  The  nodes  in 
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REVISE/UPDATE 

SmJADON 

ASSESSMENT 


CURRENT 
ASSESSMENT 
ENEMY  STATUS: 
STATIONARY  AT 
LOC  X 


LOC  X  MAIN  ATTACK 
WITHIN,  PROJECTED 
ARTILLARY  ALONG 
RANGE  CORRIDOR  A 


ENEMY:FRIENDLY 
RATIO  - 
CORRIDOR  A 
<  6:1 


TIME  LAPSE 
FROM  PREVIOUS 
SITUATION 
UPDATE  >  1  HR. 


Figure  3-6.  And/Or  Tree  of  Rules. 


a  semantic  net  stand  for  objects,  concepts  or  events.  Common  arcs  used  in 
representing  hierarchies  are  IS -A  and  HAS-PART. 

Figure  3-7  shows  a  simple  semantic  net  representation  of  "the  Abrams 
is  a  tank"  and  "a  tank  is  an  armored  vehicle."  Since  the  IS-A  relation  is 
transitive,  we  can  infer  that  the  Abrams  is  an  armored  vehicle.  Thus, 
relations  such  as  IS-A  and  HAS -PART  establish  a  property  inheritance 
hierarchy  in  a  semantic  net.  This  is  illustrated  in  Figure  2-8  which 
expands  on  the  previous  example.  In  this  case,  several  different 
bottom- level  object  nodes  can  be  traced  up  to  the  object,  "armored 
vehicle."  A  gun  turret  is  part  of  a  tank;  the  link  HAS-PART  Is  thus 
top-down.  On  the  other  hand,  "Abrams"  is  an  instance  of  the  category, 
Tank,  and  the  IS-A  link  is  bottom-up.  "Bradley  is  an  instance  of  an 
armored  personnel  carrier."  Tank  and  Armored  Personnel  Carrier  are 
upward- linked  instances  of  the  broader  category,  Armored  Vehicle.  Armored 
Vehicle  has  various  features  that  will  uniquely  identify  it.  These 
include  Engine  and  Road  Wheels,  linked  top-down  with  a  HAS-PART  connector 
from  Armored  Vehicle.  The  object  mode,  Engine,  in  turn,  HAS-PART 
Cylinders  among  its  various  constituents  (n.b.,  an  analogous  connector 
flowing  bottom-up  would  be  PART-OF) .  Semantic  nets  are  a  useful  way  to 
represent  knowledge  in  domains  that  use  well-established  taxonomies  to 
simplify  problem  solving  (Waterman,  1986). 


Figure  3-7.  Semantic  Net  Representation. 


1.  Semantic  nets  used  to  describe  natural  languages  also  use  arcs  such  as 
subject,  object,  and  relation  (Anderson,  1980). 
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Figure  3-8:  Semantic  Net  Representation  of  Bottom-Level  Object  Nodes 


Highly  Constrained  Mental  Models.  In  the  section  describing 
interpretative  approaches  to  methods  of  knowledge  structuring,  we 
introduced  a  construct  for  the  representation  of  causal  knowledge,  which 
we  called  a  mental  model.  It  was  defined  as  having:  (1)  contextual 
information;  (2)  a  description  of  the  objects  that  are  basic  to  the 
process  being  described;  (3)  the  "forces"  involved  in  the  process,  or 
medium  of  interaction  between  objects;  (4)  a  description  of  the 
interaction  between  objects  (i.e.,  forces  acting  on  objects);  and  (5) 
basic  outcomes  of  the  process. 

This  representation  format  can  be  used  not  only  for  abstract -level 
and  broadly  specified  processes,  but  for  more  concrete  and  narrowly 
specified  processes  as  well.  Figure  3-9  illustrates  a  mental-model 
representation  of  the  process  of  interception  of  electronic  signal 
communications . 

Integrated  Generative  Knowledge  Representation .  The  integrated 
generative  knowledge  representation  scheme  utilizes  all  three  of  these 
generative  approaches  to  knowledge  structuring  (i.e.,  production  rules, 
semantic  networks,  and  mental  models).  This  approach  to  the 
representation  is  essentially  production  rule-based,  however,  since 
production  rules  are  used  in,  and  are  well -suited  to,  a  wide  variety  of 
problem-solving  and  decision-making  domains.  The  integrated  approach  to 
representation  is  designed  to  capitalize  on  the  strengths  of  these  three 
different  knowledge  structures  while  minimizing  their  collective 
shortcomings.  The  strength  of  production  rule-based  representations  is 
that  they  are  action-oriented  and  lend  themselves  well  to  problem-solving 
and  decision-making  domains.  The  weaknesses  of  such  representations  is 
first,  their  inability  to  describe  the  objects  or  processes  to  which  they 
refer;  and  second,  their  inability  to  convey  an  underlying  rationale  for 
the  actions  specified  by  the  rules  when  required  conditions  are  met.  The 
first  shortcoming  can  be  addressed  by  linking  the  objects  specified  in 
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Contextual 

Information:  Battlefield  with  Electronic  Communications 

Objects:  Enemy  1  (goal:  send  message  to  E2) 

Enemy  2  (goal:  receive  message  from  El) 

Friendly  ELINT  operator  (goal:  receive  message 

from  El) 

El  electronic  communications  equipment  (El-equip.) 

E2  electronic  communications  equipment  (E2*equip.) 

FE  electronic  interception  equipment  (ELINT-equip. ) 

Forces:  Electronic  Signal 

Interaction:  El  sends  message  to  E2  using  El-equip. 

(ref:  model  of  human-elec.  comm,  equip,  interaction) 
(ref:  model  of  signal  transmission  by  elec.  comm, 
equip . ) 

E2  receives  message  from  El  using  E2- equip. 

(ref:  model  of  human-elec.  comm,  equip,  interaction) 
(ref:  model  of  signal  reception  by  elec.  comm,  equip.) 

FE  receives  message  from  El  to  E2  using  ELINT-equip. 
(ref:  model  of  human- elec,  intercept,  equip, 
interaction) 

(ref:  model  of  signal  interception  by  ELINT  equip.) 

Outcome:  E2  has  data  from  El 

FE  has  data  from  El 


Figure  3-9:  Mental  Model  of  Interception  of  Electronic  Communication 
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rules  to  a  semantic  net  of  Information  and  relationships.  The  latter 
problem  can  be  ameliorated  by  inclusion  of  mental  models.  Mental  models 
are  viewed  here  as  causal  representations  for  how  both  particular  physical 
or  conceptual  (e.g,,  arriving  at  an  insight)  events  occur.  As  such,  these 
models  can  offer  a  direct  justification  for  the  implementation  of  specific 
procedures  under  a  given  set  of  conditions.  Having  the  expert 
subsequently  perform  specific  problem  solving  would  then  serve  to  flesh 
out  the  procedures  that  he  would  actually  use  and  how  they  are  guided  by 
his  broader  objectives. 
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4.0  ELICITATION  TECHNIQUES 


4.1  Knowledge  Elicitation:  The  Pool  of  Candidate  Methods 

A  variety  of  potentially  useful  methods  of  elicitation  were  reviewed 
in  the  process  of  selecting  and  tailoring  methods  to  meet  the  elicitation 
goals  of  our  integrated  knowledge  frameworks .  Some  of  these  are  typically 
used  in  the  current  development  of  expert  systems ,  while  others  derived 
from  the  current  array  of  research  methodologies  used  in  cognitive 
psychology. 

Although  a  number  of  techniques  for  the  acquisition  of  expert 
knowledge  have  been  referred  to  in  the  literature  on  expert  systems,  there 
has  hitherto  been  little  detailed  documentation  of  their  actual  use, 
reliability,  and  relative  effectiveness.  Some  methods  are  more 
observational  in  that  the  elicitor  does  not  interrupt  or  otherwise 
interfere  with  the  expert's  problem- solving  processes,  which  are  used  to 
handle  realistic  problems.  But  most  knowledge -acquisition  techniques 
require  that  the  knowledge  engineer  play  a  more  active  role  in  an  effort 
to  get  new  facts  and  rules,  or  clarify  or  validate  existing  ones 
(Waterman,  1986). 

The  techniques  that  a  knowledge  engineer  might  typically  use  to 
extract  knowledge  from  a  domain  expert  include  (Waterman,  1986)  : 

•  on-site  observation; 

•  problem  discussion; 

•  problem  description; 

•  problem  analysis; 

•  system  refinement; 

•  system  examination; 

•  system  validation. 


On-site  observation  requires  watching  the  expert  solve  real  problems 
on  the  job.  Problem  discussion  involves  an  exploration,  in  an  interview 
between  the  elicitor  and  expert,  of  the  kinds  of  data,  knowledge,  and 
procedures  needed  to  solve  specific  problems.  Problem  description  entails 
having  the  expert  describe  a  prototypical  problem  corresponding  to  each  of 
the  basic  solution  strategies  identified  by  the  expert  as  being  commonly 
used  in  that  domain.  Problem  analysis  refers  to  methods  in  which  the 
expert  would  be  given  realistic  problems  and  be  asked  to  think  aloud  while 
the  elicitor  periodically  interrupts  this  process  to  probe  for  the 
rationale  behind  the  expert's  reasoning. 
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The  remaining  methods  require  that  a  prototype  be  developed  and  used 
as  part  of  the  elicitation  process.  System  refinement  thus  involves 
having  the  expert  supply  problems  for  an  expert-system  prototype  to  solve 
using  rules  acquired  during  earlier  elicitation  sessions.  System 
examination  entails  having  the  expert  examine  and  critique  the  prototype 
system's  rules  and  control  structures.  System  validation  is  defined 
(Waterman,  1986)  as  involving  the  presentation  of  cases  solved  by  the 
expert  and  the  prototype  system  to  other  outside  experts  for  review. 

The  use  of  a  prototype  for  "knowledge  engineering"  (i.e.,  knowledge 
elicitation  for  building  expert  systems)  is  rather  common,  and 
"rapid-prototyping"  is  a  system-development  strategy  that  has  been 
advocated  by  a  number  of  influential  Al  researchers  (Hayes -Roth,  Waterman, 
and  Lenat,  1983).  While  there  are  some  advantages,  there  are  also 
disadvantages  to  this  approach  to  knowledge  elicitation.  While  it  does 
provide  a  tangible  and  specific  example  of  the  kinds  of  abstract  goals, 
objects,  and  strategies  that  have  been  talked  about,  helps  the  elicitor 
better  comprehend  the  problem  domain,  and  provides  a  focus  for  discussions 
between  elicitor  and  expert,  the  rapid  prototype  can  also  serve  as  a 
rather  spurious  anchor  that  biases  the  direction  of  subsequent 
knowledge -elicitation  work  toward  extension  or  refinement  of  the 
prototype's  knowledge  base.  The  prototype  can  also  predetermine  the 
approach  to  knowledge  representation  that  is  taken  in  the  final  version  of 
the  system  by  virtue  of  the  availability  of  its  knowledge  representation, 
regardless  of  the  ultimate  appropriateness  of  the  approach  taken  in  the 
prototype  software. 

Knowledge  acquisition  that  emphasizes  the  use  of  a  rapid  prototype  is 
much  in  keeping  with  the  overall  approach  to  knowledge  elicitation  adopted 
by  AI  researchers,  however.  To  this  community,  "knowledge"  refers  to  the 
information  a  computer  program  needs  before  it  can  behave  intelligently 
(Waterman,  1986),  so  elicitation  focuses  on  what  computer  programs  need  to 
solve  problems  rather  than  on  what  human  experts  actually  use  to  solve 
problems . 

Approaches  to  elicitation  from  cognitive  psychology  have  the  reverse 
emphasis  since  this  field  is  primarily  concerned  with  human  knowledge  and 
how  it  is  organized,  as  well  as  its  contents.  Methods  that  are 
potentially  useful  for  expert -knowledge  elicitation  include: 

•  directed  interview 

•  memory  recall  and  reconstruction; 

•  cloze  experiments; 

•  multidimensional  scaling  and  sorting  experiments; 

•  personal  construct  methods; 

•  protocol  analysis; 

•  direct  observation. 
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Directed  interview  techniques  involve  asking  the  expert  rather  broad 
questions  about  the  problem  domain,  followed  by  questions  to  pursue  more 
detailed  description  of  the  objects  and  processes  referred  to  in  the 
expert's  earlier  responses.  This  method  is  useful  in  development  of  a 
broad  framework  for  domain  knowledge,  and  should  be  effective  in 
elicitation  of  high  level  knowledge. 

Memory  recall  and  reconstruction  techniques  can  be  applied  in  a 
variety  of  ways.  For  example,  the  elicitor  could  have  the  expert  try  to 
remember  episodes  of  his  past  problem  experiences  in  the  domain  in 
question.  The  details  of  remembered  episodes  may  inform  the  elicitor  as 
to  what  features  of  the  domain  are  important  and  what  features  organize 
memories,  and  thus  knowledge  (Anderson,  1980).  One  difficulty  with  the 
variation,  however,  is  that  the  unstructured  nature  of  experts'  responses 
may  impede  its  being  cast  in  a  more  meaningful  structure  that  essentially 
captures  the  experts'  own  representations  of  the  domain. 

Another  form  of  the  recall  and  reconstruction  method  that  would 
impose  more  constraints  on  experts'  responses  would  be  to  present  them 
with  probe  episodes  and  ask  them  to  recall  any  similar  ones .  In  this 
case,  commonalities  between  the  probe  and  the  recalled  episode  might 
indicate  how  this  information  is  organized  in  memory. 

An  even  more  structured  use  of  the  recall  and  reconstruction  method 
would  involve  experts  reading  descriptions  of  problem  episodes  relevant  to 
their  domain  and  then  trying  to  recall  what  they've  read.  Studies  have 
shown  that  people  can  best  recall  information  which  is  consistent  with  the 
knowledge  they  already  have  about  the  domain,  and  will  often  erroneously 
recall  information  which  was  not  present  in  the  material  they  just  read, 
but  is  typically  associated  with  the  case  they  have  read  about. 

Cloze  experiments  require  that  experts  either  recall  or  construct 
information.  The  recall  task  is  generally  more  constrained  than  in  the 
use  of  recall  and  reconstruction  methods.  In  cloze  experiments,  subjects 
are  given  materials  which  are  incomplete  (e.g.,  a  word  may  be  missing  in  a 
sentence  or  objects  or  events  may  be  missing  in  a  list  or  sequence)  and 
are  asked  to  complete  them.  Subjects' --in  this  case,  the  experts' -- 
choices  of  what  to  fill  in  should  reflect  what  is  anticipated  and  is  thus 
a  useful  vehicle  for  capturing  information  organized  by,  or  in  association 
with,  these  objects  (cf . ,  Bower,  Black  and  Turner,  1979;  Seifert, 

Robertson  and  Black,  1982;  Smith  and  Collins,  1981). 

Multidimensional  scaling  and  sorting  experiments  are  methods  that 
could  be  used  to  elicit  factors  that  mediate  perceived  similarity  among 
pieces  of  knowledge.  Such  methods  have  been  used  to  measure  similarity 
among  different  semantic  concepts  (cf . ,  Rips,  Schoben  and  Smith,  1973), 
and  sorting  experiments  have  been  used  to  identify  shared  themes  across 
event  sequences.  A  shortcoming  of  these  methods,  however,  is  that 
although  they  may  reveal  features  that  determine  similarity,  they  do  not 
reveal  directly  why  these  features  are  the  critical  ones,  or  what 
functions  these  features  serve  in  relation  to  other  features  of  the 
domain. 
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Personal  construct  methods  entail  elicitation  of  knowledge  in  the 
particular  form  of  "constructs"  which  represent  strongly  contrasting 
concepts  along  comparable  dimensions  (e.g.,  Kelly,  1963),  which  are 
organized  hierarchically,  like  frames  or  schemas.  As  such,  they  can  be 
elicited  using  a  variety  of  methods.  For  example,  an  elicitor  might  ask 
the  expert  to  describe  both  a  "very  good"  solution  and  a  contrasting  "very 
poor"  solution  to  a  particular  problem. 

Protocol  analysis  refers  to  the  analysis  of  think- aloud  protocols  of 
expert  problem  solving  (Newell  and  Simon,  1972).  These  protocols  are 
obtained  during  an  elicitation  session  in  which  the  expert  is  given  a 
specific  domain  problem  and  is  asked  to  "think  aloud"  as  he  goes  about 
solving  it.  Although  there  are  detractors  of  this  elicitation  approach 
who  contend  that  people  are  not  directly  aware  of,  and  cannot  accurately 
describe,  their  internal  information  processing  (Nisbett  and  Wilson, 

1977),  it  also  has  strong  supporters  (Ericsson  and  Simon,  1984)  who  argue 
that  the  concurrent  (with  the  behavior  under  investigation) ,  particular 
(i.e.,  tied  to  the  specific  task  being  done),  and  undirective  (i.e.,  the 
elicitor  does  not  interrupt  or  try  to  influence  the  course  of  problem 
solving)  nature  of  this  elicitation  method  offers  the  best  available  means 
for  constructing  accurate  models  of  expert  knowledge. 

Direct  observation  is  the  least  obtrusive  and  thus,  perhaps,  the  most 
natural  approach  to  knowledge  elicitation.  Employing  this  method,  the 
elicitor  would  give  the  expert  a  realistic  problem,  or  observe  the  expert 
solving  problems  in  "real-world"  task  environments.  Although  this  method 
assures  that  the  process  of  elicitation  does  not  in  any  way  affect  the 
expert's  problem-solving  behavior,  it  has  the  corresponding  drawback  of 
forcing  the  elicitor  to  make  very  substantial  inferences  about  the 
expert's  knowledge  and  precisely  how  it  is  used  to  direct  the  behavior 
being  observed.  This  method  is  most  effective  when  there  are  other  domain 
experts,  including  perhaps  the  person  doing  the  elicitation,  who  can 
provide  a  fairly  accurate  account  of  the  expert's  actions. 

Many  of  the  elicitation  methods  used  by  knowledge  engineers  could 
actually  serve  to  tap  the  same  object  and  process  knowledge  as  the  methods 
from  cognitive  psychology,  and  may  thus  suit  the  data  collection  needs  of 
this  project.  What  is  quite  different  is  the  way  elicited  knowledge  would 
be  used  by  knowledge  engineers.  Most  expert  system  builders  will  encode 
data  from  knowledge  elicitation  into  forms  that  are  compatible  with  their 
system  architecture's  predefined  requirements.  By  contrast,  we  propose 
that  knowledge  elicitation  proceed  in  ways  that  are  compatible  with  how 
the  expert  represents  them. 

The  same  techniques  might  thus  be  used  to  serve  two  distinct  elicita¬ 
tion  perspectives  each  with  a  different  flow  of  control  in  elicitation. 

The  system  builders'  approach  to  elicitation  involves  (1)  determination  of 
system  requirements  based  on  a  preselected  structure  for  knowledge  repre¬ 
sentation;  (2)  elicitation  of  the  expert  knowledge;  and  (3)  coding  the 
knowledge  into  the  system’s  representation.  The  alternative  approach  that 
we  explore  involves  (1)  determination  of  the  knowledge  an  expert  would  use 
to  perform  the  task;  (2)  selection  of  the  elicitation  methods  compatible 
with  an  expert's  approach;  (3)  development  of  models  of  the  expert's 
knowledge;  and  (4)  evaluation  of  the  quality  of  these  models  and  methods. 
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4.2  Interpretative  and  Generative  Methods  of  Knowledge  Elicitation 

From  the  pool  of  elicitation  techniques  described  above,  techniques 
can  be  packaged  to  form  a  methodology  of  knowledge  elicitation  that 
assumes  an  underlying  model  of  knowledge  representation  and  expert  problem 
solving.  Below  we  offer  a  demonstration  of  two  such  methodologies.  One 
assumes  that  expert  knowledge  and  problem  solving  is  organized  in  a 
■top-down"  or  interpretative  fashion,  while  the  other  assumes  that  expert 
knowledge  and  problem  solving  is  organized  in  a  "bottom-up"  or  generative 
fashion.  Additional  techniques  can  be  used  to  supplement  the  primary 
techniques  of  the  methodologies. 

It  should  be  noted  that  in  constructing  these  methodologies,  two 
principles  are  operating;  first,  the  first  half  of  the  elicitation  session 
is  structured  and  constrains  the  expert's  responses,  while  the  second  half 
is  less  structured  (once  the  framework  has  been  set).  Second,  the  first 
technique  used  on  the  expert  is  geared  toward  eliciting  knowledge  at  the 
level  of  abstraction  that  the  method  assumes  will  be  the  organizing 
principle.  Specifically,  the  interpretative  method  starts  with  a 
technique  which  is  designed  to  obtain  high-level,  general  knowledge  (i.e., 
interview) ,  while  the  generative  method  starts  with  a  technique  which  is 
designed  to  obtain  low-level,  specific  knowledge  (i.e.,  problem  solving). 
As  the  elicitation  session  proceeds,  the  techniques  change  to  induce  a 
"top-down"  or  "bottom-up"  flow  of  knowledge  for  the  interpretative  and 
generative  method,  respectively.  The  initial  structure  placed  on  the 
expert  is  designed  to  "force"  the  expert  into  an  interpretative  or 
generative  mode. 

By  forcing  the  experts  into  one  of  these  modes,  the  elicitor  can 
control  and  insure  that  each  method  is  inducing  a  corresponding  knowledge 
use  and  flow  of  knowledge  control  from  the  experts.  Once  these  modes  have 
been  thrust  on  the  experts,  the  elicitor  can  compare  interpretative  and 
generative  methods  of  knowledge  elicitation  for  their  efficacy. 

4.2.1  Interpretative  method.  An  assumption  underlying  our 
interpretative  method  is  that  expert  knowledge  is  organized  in  a 
"top-down,"  hierarchical  fashion.  By  this  we  mean  that  knowledge  is 
organized  around  high-level  goals  and  plans,  which  in  turn  index  more 
low-level  procedural  and  content  knowledge.  This  can  be  viewed  as 
analogous  to  deductive  reasoning,  where  reasoning  proceeds  from  general  to 
specific  cases. 

Our  interpretative  elicitation  method  prescribes  that  knowledge 
elicitation  should  occur  in  an  interpretative  method,  i.e.,  by  first 
eliciting  general  knowledge,  primarily  focusing  on  goal  and  planning 
knowledge  and  successively  refining  this  knowledge  until  specific 
procedures  and  content  knowledge  are  elicited.  Two  knowledge  elicitation 
techniques  form  the  heart  of  this  method:  structured  interview  and 
"top-down"  problem  solving. 

Structured  interview .  A  structured  interview  is  a  question  and 
answer- type  procedure  where  the  elicitor  asks  the  expert  a  series  of 
questions  and  follow-up  questions  which  are  specifically  designed  to  build 
certain  knowledge  structures.  Questions  are  therefore  guided  by  the 
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specific  structures  the  elicitor  is  trying  to  build  and  the  slots  that  are 
contained  within  those  structures. 

While  a  structured  interview  is  a  general  technique  which  can  be 
adapted  to  elicit  knowledge  concerning  a  variety  of  knowledge  structures, 
we  discuss  how  this  technique  was  used  in  the  interpretative  method. 
However,  knowledge  elicitors  could  readily  adapt  the  interpretative 
version  of  the  technique  for  elicitation  methodologies. 

In  the  interpretative  method,  the  structured  interview  is  used  to 
elicit  script-based  and  supporting  knowledge  (i.e.,  frames  and  mental 
models  which  supplemented  the  script-based  knowledge).  As  such,  this 
induces  experts  to  use  their  knowledge  in  an  interpretative  fashion,  since 
the  elicitor  largely  directs  what  knowledge  the  expert  provides  and  when 
(as  will  seen  below).  In  order  to  accomplish  this,  the  elicitor  takes  his 
script  template  and  uses  each  slot  as  a  basis  for  generating  questions. 

The  details  for  doing  this  are  outlined  below. 

The  elicitor  designates  which  initial  script  he  is  going  after  by  the 
domain  area  (e.g.,  situation  development,  order  of  battle).  For  the  sake 
of  illustration  we  will  assume  that  the  elicitor  is  interested  in 
knowledge  pertaining  to  situation  development.  The  first  slot  on  the 
script  template  is  tracks,  i.e.,  major  conditions  which  dictate  variations 
in  the  way  the  script  is  executed.  Hence,  the  elicitor  begins  by  asking 
the  expert  if  there  are  any  conditions  which  might  influence  they  way 
situation  development  is  carried  out.  If  the  elicitor  is  able,  and  the 
expert  is  having  trouble  either  understanding  the  question  or  coming  up 
with  an  answer,  he  might  prompt  the  expert  by  giving  tentative  examples. 
These  examples  should  be  general  enough  so  as  not  to  bias  the  expert  too 
much,  but  sufficiently  relevant  to  be  a  useful  prompt.  In  this  case,  a 
useful  prompt  might  be  (and  one  that  was  used  in  the  present  knowledge 
elicitation),  "for  example  differences  in  echelon  or  level  of  conflict?". 

Once  the  elicitor  has  specified  what  the  different  tracks  are,  the 
elicitor  should  get  examples  of  each  (for  example,  if  situation 
development  is  different  depending  upon  level  of  conflict,  then  the 
elicitor  should  get  examples  of  the  different  levels  of  conflict,  e.g., 
high  intensity  vs.  low  intensity  conflicts).  The  elicitor  should  then  ask 
the  expert  to  describe  how  situation  development  is  different  based  on 
these  different  tracks.  This  question  may  lead  to  knowledge  that  is  more 
appropriately  associated  with  other  slots  in  the  script.  However,  it 
gives  the  expert  a  chance  to  present  knowledge  that  may  be  stored  at  this 
more  general  level  as  opposed  to  stored  with  the  specific  slots. 

Once  the  elicitor  has  exhausted  his  questions  regarding  tracks,  he 
proceeds  to  the  next  set  of  questions.  Between  the  tracks  slot  and  the 
slots  holding  information  regarding  scenes  and  actions,  there  are  slots 
that  are  basically  containers  of  background  and  contextual  knowledge. 

These  slots  Include  goals  (e.g.,  the  goals  or  objectives  behind  doing 
situation  development) ,  the  entry  conditions  (conditions  which  trigger 
execution  of  the  script) ,  roles  (the  people  that  are  relevant  to  the 
script  either  as  participants  or  because  they  interact  with  the  script  in 
some  meaningful  way),  props  (the  tools  and  objects  that  are  relevant  to 
the  script),  and  outcomes  or  results  (the  state  of  the  world  that  results 
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when  the  script  is  executed,  particularly  how  it  relates  to  the  goals  of 
script,  i.e.,  which  goals  get  satisfied  and  which  goals  do  not). 

For  each  of  these  slots,  the  elicitor  asks  an  appropriate  question, 
e.g.,  "What  goals  or  objectives  are  you  trying  to  achieve  when  you  perform 
situation  development?”.  The  elicitor  should  be  prepared  to  rephrase 
these  questions  in  case  the  expert  is  unclear  as  to  what  the  elicitor 
means  or  is  interpreting  them  in  the  wrong  way.  For  example,  the  elicitor 
may  want  to  say  "Who  are  the  relevant  people  involved  or  with  whom  do  you 
interact?"  for  the  roles  slot.  Of  course,  in  trying  to  clarify  the 
question,  the  elicitor  should  be  careful  not  to  constrain  the  experts 
answer.  For  example,  the  relevant  roles  of  the  script  might  go  beyond 
those  that  the  expert  interacts  with.  Therefore,  after  the  expert  gives 
his  answer,  the  elicitor  should  be  prepared  to  ask  a  follow-on  question 
such  as  "Is  there  any  one  else  who  affects  what  you  do,  even  if  you  don't 
interact  directly  with  them?".  If  the  elicitor  could  give  some  examples, 
even  if  they  are  wrong,  it  may  help  to  prompt  the  expert,  e.g.,  "For 
example,  the  Corps  commander  or  the  enemy  forces?". 

Below,  we  present  sample  questions  for  these  slots  along  with 
alternative  ways  of  asking  them  and  generating  follow-ups. 

Goals:  "What  are  your  goals  and  objectives  in  doing  situation 

development?"  "Why  do  you  do  situation  development?"  "What  are  you 
trying  to  accomplish  or  achieve?" 

Entry  Conditions:  "What  conditions  trigger  your  doing  situation 
development?"  "When  do  you  do  situation  development?"  "Is  it  an 
ongoing  process  or  does  something  occur  to  trigger  the  process,  like 
a  change  in  the  battle  or  an  order  from  your  commander?"  "Is  there 
any  other  set  of  circumstances  that  might  trigger  it?" 

Roles:  "What  people  play  a  role  in  doing  situation  development?" 
"With  whom  do  you  interact?"  "Are  there  any  people  you  don't 
interact  with  but  still  influence  what  you  do,  like  the  enemy 
forces?" 

Props:  "What  kind  of  tools  or  objects  do  you  work  with?"  "If  I  were 
to  walk  into  your  shop,  what  kinds  of  things  might  I  find  there  or 
see  you  working  with?”  "Do  you  have  things  like  maps,  reference 
books  and  workbooks?" 

Outcomes:  "When  you've  completed  situation  development,  what  have 
you  accomplished?"  "What  outcomes  occur  as  a  result  of  doing 
situation  development?"  "What  do  you  know  as  a  result  of  doing 
situation  development  that  you  didn't  know  before?" 

As  these  questions  are  being  asked,  it  is  useful  to  tie  them  to  the 
particular  tracks  that  have  been  laid  out  before.  Hence,  the  elicitor 
might  ask  questions  like,  "In  a  low  Intensity  conflict  are  the  players  any 
different  than  in  a  high  intensity  conflict?"  or  "Do  you  work  with 
different  tools  at  battalion  level  than  at  division  level?",  etc. 
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It  should  be  noted  that  during  the  elicitation,  the  expert  will 
undoubtedly  make  references  to  concepts  or  objects  that  are  relevant  to 
the  script.  Such  references  are  probably  most  likely  to  occur  when 
eliciting  knowledge  regarding  roles  and  props.  For  example,  in  describing 
the  enemy  forces ,  the  expert  may  make  reference  to  "motorized  rifle 
division".  When  such  references  occur,  the  elicitor  must  know  what  they 
are  and  what  their  role  in  the  script  is .  In  order  to  answer  these 
questions  they  elicitor  should  attempt  to  elicit  an  object  frame  about 
that  concept.  The  method  for  doing  so  is  analogous  to  the  method  fo*. 
eliciting  scripts.  The  elicitor  takes  the  general  template  for  an  object 
frame  and  uses  each  slot  as  a  basis  for  generating  questions.  For 
example,  for  the  "ISA"  slot,  the  elicitor  might  ask,  "Is  there  a  more 
general  category  that  (e.g.)  a  motorized  rifle  division  is  an  example  of?" 

Once  the  background  knowledge  has  been  elicited,  the  elicitor  then 
focuses  on  the  heart  of  the  script,  namely  the  scenes  and  the  actions. 

The  boundaries  between  them  may  not  always  be  clear.  In  section  7,  we 
discuss  how  to  distinguish  between  them.  Proactively,  an  elicitor  may 
attempt  to  elicit  information  that  will  help  him  decide  later  what  the 
scene  boundaries  are  (e.g.,  information  regarding  the  setting  associated 
with  different  actions,  the  people  involved  and  the  objectives  in  relation 
to  the  overall  script)  by  asking  questions  designed  to  get  at  such 
information,  e.g.,  "Where  are  you  when  you're  doing  this?",  "Who  is  with 
you?",  "What  is  the  purpose  of  doing  this?",  and  "Is  this  action  and  the 
previous  one  part  of  the  same  procedure?". 

In  order  to  elicit  the  action  relevant  knowledge  that  will  fill  in 
the  script,  the  elicitor  asks  the  expert  to  give  a  step-by-step  outline  of 
the  procedures  he  follows  in  performing  the  script.  After  the  expert  goes 
through  the  process,  the  elicitor  can  recursively  ask  the  expert  to 
provide  additional  details,  to  fill  in  specific  actions.  For  example,  if 
the  expert  is  describing  situation  development  and  says  "First,  I  get  a 
briefing  from  the  officer  that  I  am  relieving.  Then  I  do  a  terrain 
analysis.  Then  I  look  at  the  order  of  battle  data,  etc.",  the  elicitor 
may  wait  until  the  expert  is  through  and  ask  questions  like,  "How  do  you 
do  the  terrain  analysis?".  After  the  expert  describes  that,  the  elicitor 
can  repeat  the  process,  e.g.,  "How  do  you  determine  what  blocking  terrain 
is?",  "How  do  you  decide  if  the  river  is  fordable?",  and  so  on  until  the 
expert  cannot  specify  the  knowledge  in  further  detail. 

While  the  elicitor  is  eliciting  the  scene  and  action  knowledge,  he 
may  wish  to  elicit  mental  model -related  knowledge  which  explains  why  the 
particular  actions  are  being  performed.  These  models  will  probably  link 
actions  to  the  scene-related  goals  and  scenes  with  script-related  goals. 
For  example,  the  action  of  marking  blocking  terrain  on  the  map  serves  the 
scene  (terrain  analysis)  related  goal  of  determining  where  the  enemy  can 
and  cannot  go.  The  terrain  analysis  scene  serves  the  situation 
development  script  goal  of  inferring  what  the  enemy  course  of  action  will 
be  (where  the  terrain  analysis  provides  inputs  of  what  enemy  courses  of 
action  the  terrain  can  support) . 

We  advocate  a  two-step  process  for  eliciting  mental  model-related 
knowledge.  First,  the  elicitor  can  ask  the  expert  a  "Why?"  question 
regarding  the  procedure  he  is  describing.  The  expert  is  then  allowed  to 


38 


describe  in  an  open-ended  format  the  reasons  for  the  procedure.  The 
elicitor  tries  to  supplement  this  answer  by  asking  questions  specifically 
designed  to  elicit  knowledge  relevant  to  the  slots  of  a  mental  model.  For 
example,  the  expert  might  be  asked,  "What  are  the  objectives  behind  this?" 
or,  in  "What  context  would  this  occur"  (a  variant  of  this,  to  get  at  the 
level  of  generality,  might  be  "Is  this  a  universal  principle  or  does  it 
apply  only  to  specific  cases?"),  etc.  The  elicitor  can  focus  the  expert 
on  particular  goals  by  asking  questions  like,  (e.g.)  "How  does  doing 
terrain  analysis  help  you  to  predict  the  enemy  course  of  action?".  In 
fact,  in  principle,  the  elicitor  can  relate  any  action  to  any  goal  in  the 
script  and  elicit  a  mental  model  for  it.  The  elicitor  must  decide  to  what 
extent  to  pursue  this. 

The  above  is  a  description  of  the  structured  interview  technique. 
There  are  some  general  issues  the  elicitor  must  keep  in  mind.  First,  the 
elicitor  must  decide  in  what  depth  to  pursue  the  expert's  knowledge. 

Since  this  technique  can  be  used  recursively,  the  elicitor  could  keep 
going  until  the  expert  cannot  answer  or  gets  annoyed.  The  elicitor  must 
decide  what  objectives  he  has  and  how  this  affects  the  degree  of  detail 
necessary  to  be  elicited. 

In  order  to  make  a  judgment  on  this,  the  elicitor  should  consider  the 
following  trade-offs:  forcing  the  expert  to  elaborate  on  everything  may 
distract  his  train  of  thought  and  sacrifice  some  of  the  points  the  expert 
was  trying  to  make.  For  this  reason,  it  is  recommended  that  the  elicitor 
waits  until  the  expert  exhausts  what  he  was  going  to  say  and  then  go  back 
to  the  point  where  more  depth  is  desired.  If  possible,  we  advocate  that 
the  elicitor  pursue  a  train  of  thought  in  depth  before  proceeding  to  the 
next  train  of  thought,  rather  than  covering  all  the  topics  at  the  same 
level  of  detail  and  then  proceeding  to  the  next.  To  rephrase  this,  we 
advocate  eliciting  in  depth  first  rather  than  breadth  first,  unless  the 
expert  spontaneously  gives  knowledge  in  a  breadth- first  fashion. 

Second,  the  elicitor  should  try  to  safeguard  against  missing 
knowledge  that  the  expert  has.  While  it  is  probably  impossible  to  insure 
against  some  misses,  there  are  some  steps  that  the  elicitor  can  take  to 
try  to  get  a  complete  elicitation.  First,  the  elicitor  should  try  to 
avoid  interrupting  the  expert  and  allowing  him  to  finish  his  answer  before 
proceeding  to  the  next  question.  If  in  doubt,  the  elicitor  should  ask  the 
expert  if  he's  finished.  Second,  the  elicitor  can  ask  questions  like,  "Is 
there  anything  else  you  can  think  of?,"  "Is  there  some  other  situation 
where  things  might  be  different?".  If  the  elicitor  can  actually  generate 
some  sample  situations  this  may  be  helpful. 

When  the  elicitor  has  completed  the  interview  to  his  satisfaction,  he 
is  ready  to  proceed  to  the  top-down  problem  solving  technique. 

Top-down  Problem  Solving.  Top-down  problem  solving  is  a  technique 
which  is  very  useful  in  a  supplemental  capacity  to  an  interview  technique, 
particularly  a  structured  interview.  In  a  capsule,  this  technique 
involves  giving  experts  problems  to  work  on  and  observing  and  recording 
their  performance.  The  elicitor  is  particularly  interested  in  comparing 
how  the  expert  actually  solves  a  concrete  problem  to  how  he  describes  the 
problem  solving  approach  in  the  more  abstract  interview.  Hence,  the 
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elicitor  should  be  prepared  to  ask  questions  to  link  the  two  techniques 
together  and  to  integrate  the  knowledge  produced  by  the  problem  solving 
technique  with  the  knowledge  produced  by  the  interview. 

The  top-down  problem  solving  technique  is  more  unstructured  than  the 
structured  interview.  As  such,  this  technique  should  not  bias  the  expert 
to  use  his  knowledge  in  an  interpretative  or  generative  fashion.  In  order 
to  conduct  this  technique,  the  elicitor  must  have  available  a  problem 
which  is  similar  to  the  kind  that  the  expert  typically  works  on.  This 
will  include  all  the  background  information  as  well  as  the  tools  and 
reference  materials  that  the  expert  will  need  to  solve  the  problem. 

Hence,  this  technique  requires  much  preparation.  If  the  expert  is  used  to 
having  human  resources  available  when  solving  the  problem,  it  will  be  very 
useful  to  have  such  resources  available.  This  can  be  done  in  several 
ways.  If  possible,  the  expert  can  solve  the  problem  in  a  group  setting 
with  the  other  group  members  playing  the  roles  of  the  people  the  expert 
normally  works  with.  This  is  probably  the  most  difficult  solution  to 
arrange.  If  the  elicitor  is  sufficiently  expert  in  the  subject  matter 
area,  he  can  play  the  role  of  the  additional  people.  This  may  be 
undesirable  if  the  additional  people  would  normally  be  co-problem  solvers 
as  opposed  to  repositories  of  information,  as  participation  would  probably 
distract  the  elicitor  from  conducting  effective  knowledge  elicitation. 

A  third  alternative,  possibly  the  best  one,  would  be  for  the  elicitor 
to  have  a  subject  matter  consultant  who  is  familiar  with  both  the  domain 
in  question  and  the  elicitor' s  objectives.  Such  a  person  could  provide 
the  human  resources  that  the  elicitor  needs  in  a  way  that  won't  obstruct 
the  elicitation  process.  If  the  elicitor  lacks  subject  matter  expertise, 
then  it  is  important  that  a  separate  subject  matter  expert  serves  as  the 
co-elicitor  rather  than  having  the  subject  himself  play  that  role.  First, 
having  the  subject  participate  as  an  elicitor  (e.g.,  trying  to  anticipate 
when  explanations  are  going  to  be  necessary  or  determining  the  level  of 
complexity  to  focus  on)  may  disrupt  the  expert's  natural  reasoning 
processes  and  spoil  the  elicitation  session.  Second,  if  the  subject 
perceives  that  the  elicitor  is  naive,  he  may  discuss  topics  at  an 
elementary  level  rather  than  at  an  expert  level.  Having  a  subject  matter 
expert  as  a  co-elicitor  may  encourage  the  subject  to  speak  at  expert 
levels.  Other  advantages  of  the  subject  matter  consultant  are  that  he  can 
provide  the  background  to  the  problem  and  answer  clarifying  questions,  and 
also  having  such  a  consultant  may  put  the  expert  at  ease  (i.e.,  the  expert 
feels  that  there  is  one  of  his  "kind"  there)  and  may  add  credibility  to 
the  elicitor  in  the  eyes  of  the  expert. 

The  expert  is  given  an  overview  to  the  problem  and  asked  to  solve  it. 
The  elicitor  should  be  precise  on  what  the  expert's  goals  should  be  (e.g., 
"Give  a  briefing  on  where  you  think  the  enemy  main  attack  will  come.")  and 
be  informed  as  to  what  resources  the  expert  can  use  and  will  have 
available.  Since  problem  solving  is  often  a  basis  for  performance,  it  is 
important  to  assure  the  expert  that  he  is  not  being  evaluated  and  that  the 
elicitor  is  interested  in  the  expert's  process,  not  the  correctness  of  his 
answer . 

The  primary  source  of  data  will  be  inferences  drawn  from  observation 
and  having  the  expert  explain  what  he  is  doing.  Hence,  the  elicitor 
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should  ask  the  expert  to  think  aloud  as  he  is  working  on  the  problem  and 
explain  what  he  is  doing.  The  expert  should  be  told  that  the  emphasis  is 
on  process  and  that  expert  should  try  to  articulate  the  process  as  well  as 
possible.  It  is  important  to  note  that  not  all  experts  will  feel 
comfortable  doing  this,  since  most  experts  are  probably  used  to  working 
silently  and  may  not  be  used  to  explaining  problem  solving  processes  to 
others  (for  this  reason  it  may  be  useful  to  screen  experts,  e.g.,  find 
some  who  have  taught  the  material  and  hence  are  used  to  explaining  the 
process).  If  this  is  the  case,  the  elicitor  may  have  to  play  an  active 
role  in  trying  to  get  che  expert  to  talk.  One  way  to  do  this  is 
periodically  to  urge  the  expert  to  verbalize,  particularly  if  he  is  going 
for  periods  of  time  without  speaking. 

Another  method  is  useful,  particularly  if  the  expert  is  having 
trouble  simultaneously  articulating  and  working  on  the  problem  (this  would 
be  indicated  by  long  periods  of  silence  where  prompting  does  little  good) . 
This  method  involves  stopping  the  expert  periodically  and  asking  him  to 
describe  what  he  had  been  doing.  We  have  found  that  this  method  works 
very  well.  It  is  important,  however,  to  stop  the  expert  after  he  has 
completed  a  sub- task,  i.e.  seems  to  have  paused  or  is  changing  procedures. 
Examples  of  this  might  be  after  the  expert  has  stopped  analyzing  a  map, 
after  he  has  stopped  reading  a  manual,  etc.  It  is  important  to  try  not  to 
break  the  expert's  train  of  thought  while  he  is  working  on  the  problem. 

In  general,  the  elicitor  provides  less  direction,  so  as  not  to 
interfere  with  how  the  expert  wants  to  solve  the  problem.  The  elicitor 
should  restrict  his  questions  to  those  of  a  clarifying  nature  (e.g., 

"Could  you  explain  what  you  just  did  there?")  or  trying  to  link  what  the 
expert  is  doing  to  what  he  said  earlier  in  the  structured  interview  (e.g., 
"Are  you  doing  the  terrain  analysis  that  you  described  in  the 
interview?").  The  elicitor  should  try  to  ascertain  how  the  expert  is 
solving  the  problem.  To  do  this,  he  may  need  to  ask  questions  like  "What 
information  are  you  looking  for/using?",  "What  hypotheses  have  you 
developed?".  Such  questions  may  relate  back  to  answers  the  elicitor  gave 
earlier  (e.g.,  "Earlier  you  thought  the  enemy  might  be  attacking  in  the 
north.  How  does  this  new  evidence  relate  to  that  conclusion?”). 

The  elicitor  must  determine  the  timing  of  particular  questions. 
Questions  that  may  lead  to  general  discussions  are  probably  best  left  for 
periods  reserved  for  summaries  of  what  the  expert  has  been  doing. 

However,  specific  questions  like  "What  information  are  you  looking  for?" 
need  to  be  timed  to  when  the  expert  is  actually  looking  for  it. 

4.2.2  Generative  method.  Contrasting  with  the  interpretative  method 
of  knowledge  elicitation  is  the  generative  method,  which  is  based  on  the 
assumption  that  knowledge  is  organized  in  a  "bottom-up"  or  generative 
fashion.  By  this  we  mean  that  knowledge  is  organized  around  low-level 
situation  specific  rules  and  content  knowledge  which  gets  accumulated  from 
concrete  experiences  and  is  then  generalized  to  higher  levels  of 
abstraction.  The  principle  containers  of  such  knowledge  might  be 
production  rules,  semantic  nets  and  mental  models  which  capture  the  how, 
what  and  why  aspects  of  knowledge,  respectively. 


41 


The  basic  philosophy  behind  the  generative  method  is  to  try  to 
capture  knowledge  that  centers  around  specific  procedures  (both  for 
executing  actions  and  making  inferences)  and  specific  objects  and/or 
concepts.  As  this  knowledge  is  accumulated,  the  elicitor  attempts  to  link 
the  individual  pieces  of  knowledge  together  by  chaining  and  also  attempts 
to  generalize  the  knowledge  by  eliciting  higher  level  rules  and  more 
general  categories  of  concepts  and  objects.  Hence,  a  complete  knowledge 
base  is  built  from  "the  bottom  up". 

Like  the  interpretative  method,  the  generative  method  has  at  its 
heart  two  primary  techniques,  interview  and  problem  solving.  However, 
these  techniques  have  been  adapted  for  use  in  a  generative  strategy  of 
knowledge  elicitation.  Similarly,  the  order  of  administration  has  been 
reversed  with  problem  solving  occurring  first  and  interview  acting  as  a 
supplemental  technique.  This  ordering  relates  back  to  our  notion  that 
problem  solving  as  a  technique  is  best  suited  for  eliciting  specific  and 
more  low-level  knowledge  while  interview  as  a  technique  is  best  suited  for 
eliciting  general  and  more  high-level  knowledge. 

We  discuss  our  generative  problem  solving  and  interview  techniques 
below.  We  should  note  that  in  practice  these  techniques  are  not  as 
discrete  as  in  the  interpretative  method.  Rather  the  techniques  overlap 
with  problem  solving  being  emphasized  in  the  initial  stage  of  the 
elicitation,  but  once  a  significant  knowledge  base  has  been  built  up,  the 
problem  solving  technique  gets  phased  out  while  the  interview  technique  is 
phased  in  to  attempt  to  build  generalities  and  linkages  from  the  lower 
level  knowledge. 

Data-driven  problem  solving.  This  problem  solving  technique  has  been 
developed  for  the  generative  method  (Leddo  and  Cohen,  1987).  It  is  one 
which  induces  the  expert  to  solve  a  problem  in  a  generative  manner  ( i . e . , 
the  flow  of  control  of  the  expert's  problem  solving  is  made  to  be 
generative) .  The  purpose  behind  inventing  the  technique  was  the 
observation  that  people  very  often  have  unspoken  assumptions  uiey  make 
that  influence  their  judgments  and  conclusions.  Such  assumptions  are 
typically  derived  from  the  context  the  problem  is  in  and  can  mislead  an 
elicitor's  perception  as  to  how  the  expert  is  reasoning.  Even  asking  an 
expert  to  state  his  assumptions  is  often  inadequate  since  an  expert  may 
not  realize  all  the  assumptions  he  is  making  or  take  many  of  them  for 
granted  and  not  bother  listing  them. 

The  data-driven  problem  solving  technique  tries  to  control  for  these 
background  assumptions  as  much  as  possible.  Hence,  the  data-driven 
problem  solving  technique  requires  careful  control  of  the  information  that 
the  expert  has  available.  We  realize  that  it  is  impossible  to  completely 
control  for  all  assumptions  experts  can  bring  to  bear. 

The  data-driven  problem  solving  technique  requires  that  the  elicitor 
has  a  problem  available,  one  that  has  a  lot  of  data,  which  when  taken 
together,  can  lead  the  expert  to  some  solution  (although  a  more  limited 
use  of  this  technique  can  be  achieved  by  having  random  pieces  of  data) . 
Hence,  this  technique  is  most  useful  in  problems  or  domains  that  lend 
themselves  to  building  solutions  to  problems  from  individual  pieces  of 
data.  The  technique  begins  by  giving  an  expert  a  single  piece  of  datum, 
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which  could  be  in  the  the  form  of  a  message  or  a  spot  report  that  he  might 
see  while  working  at  his  job.  No  other  information  is  provided,  i.e.,  the 
expert  does  not  know  where  he  is  fighting,  at  what  echelon,  who  the  enemy 
is,  what  day  of  the  war,  etc.  Admittedly,  this  is  highly  artificial  in 
that  the  expert  would  have  extensive  background  information  in  a  realistic 
situation.  However,  the  point  of  this  technique  is  not  to  have  such 
background  information  color  the  expert's  answer. 

After  the  expert  has  looked  at  that  information,  he  is  asked  four 
basic  questions:  "What  does  this  data  mean  to  you?",  "What  assumptions 
or  conclusions  have  you  reached  as  a  result  of  this?",  "Can  you  link  this 
information  with  other  information  that  you  have  seen?  (assuming  the 
expert  has  seen  other)",  "If  you  had  additional  information,  what  would  it 
help  you  to  conclude?".  While  it  may  seem  as  though  an  expert  would  have 
great  difficulty  answering  such  questions,  as  can  be  seen  from  the 
annotated  protocol  presented  in  section  7,  experts  are  able  to  give  a  lot 
of  information  with  limited  data  to  go  on. 

The  first  question,  what  the  data  means  to  the  expert,  is  basically 
designed  to  get  a  dump  of  all  the  inferences  that  the  expert  can  generate. 
Hence,  this  question  elicits  knowledge  in  breadth  what  the  expert  knows 
that  relates  to  the  data.  Our  experience  with  this  technique  suggests 
that  experts  are  able  to  give  quite  a  lot  of  knowledge  even  with  such 
relatively  little  prompting  (and  in  spite  of  the  fact  that  the 
presentation  of  the  data  is  in  a  format  that  is  unusual  for  them) . 

The  second  question,  what  assumptions  and  conclusions  the  expert  has 
reached,  is  designed  to  elicit  the  background  knowledge  that  the  expert 
brings  to  bear  on  his  problem  solving.  This  type  of  question  is  useful  in 
seeing  how  the  expert  is  setting  up  the  problem  to  solve,  forming 
hypothesis  and  seeing  what  knowledge  structures  that  organize  the  data 
presented. 

The  third  question,  which  links  the  current  data  to  previous  data  and 
conclusions,  is  designed  to  get  at  the  integrative  or  truly  "generative" 
processes  that  the  expert  uses  to  put  together  the  assessment  of  the 
situation  and  arrive  at  a  solution.  Responses  to  this  question  give 
indications  of  what  data  the  experts  group  together,  what  types  of 
intermediate  conclusions  the  expert  reaches  and  how  smaller  sets  of  data 
fit  into  successively  larger  sets. 

The  fourth  question,  what  further  data  the  expert  would  like  to  have, 
is  designed  to  get  at  the  priorities  the  expert  has  in  terms  of 
information  requirements  as  well  as  what  hypotheses  the  expert  is 
developing.  A  crucial  follow-up  question  to  the  expert's  response  is  why 
the  expert  wants  that  data. 

Here,  the  elicitor  is  actively  trying  to  see  where  the  expert  is 
going  in  his  problem  solving.  This  is  an  important  issue  since  the 
structure  of  this  technique  forces  the  expert  to  follow  a  specific  path  of 
data  processing  (since  the  elicitor  controls  the  presentation  of  data) . 

It  is  important  for  the  elicitor  to  assess  what  the  expert  would  do  if 
given  the  choice  to  pursue  the  problem  in  his  own  way.  We  justify  having 
the  elicitor  control  the  data  presentation  (although  clearly  the  technique 
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could  be  run  wieh  the  expert  controlling  the  data  he  receives)  by  noting 
that,  in  a  realistic  environment,  very  often  the  data  will  come  in  a 
seemingly  random  fashion  as  a  variety  of  collection  and  analysis  assets 
will  be  reporting  back  information  as  they  can,  even  if  it  is  not  in  the 
expert's  preferred  order. 

After  the  expert  has  responded  to  these  questions,  the  elicitor 
iterates  to  the  next  piece  of  data  and  the  cycle  repeats  until  the  supply 
of  data  is  exhausted.  At  the  point  at  which  the  data  is  exhausted,  the 
elicitor  can  ask  the  expert  what  additional  piece  of  data  he  would  most 
like  to  have.  The  elicitor  can  then  make  up  that  data,  if  he  has  the 
expertise  to  do  so,  have  a  subject  matter  expert  consultant  make  up  the 
data  or  ask  the  expert  himself  to  make  up  the  data  and  proceed  with  it. 
This  latter  method  will  allow  the  expert  to  develop  his  hypotheses  better, 
which  may  have  the  advantage  of  seeing  how  that  is  done  and  have  the 
disadvantage  that  it  may  not  challenge  the  expert  to  develop  competing 
hypotheses  (unless  the  elicitor  instructs  the  expert  to  do  so) . 

Unstructured  interview.  As  the  expert  is  progressing  through  the 
problem  solving  technique,  the  elicitor  gradually  phases  in  more  and  more 
interview- type  questions.  Here,  the  interviewer  is  trying  to  clarify  what 
the  expert  is  doing,  arrive  at  generalizations  that  can  be  abstracted  from 
what  the  expert  is  doing,  and  to  force  the  expert  to  think  more  broadly 
about  the  problem  and  consider  more  alternatives. 

In  terms  of  clarification,  the  elicitor  may  ask  the  expert  to 
summarize  what  he  has  been  doing,  explain  the  rationales  behind  It,  and 
clarify  things  that  the  elicitor  does  not  understand  (presumably  as 
problem  solving  is  progressing,  the  expert  is  doing  less  of  a  "dump"  on 
pieces  of  data  and  more  complex  reasoning,  which  may  require  clarification 
for  the  elicitor) .  One  type  of  clarification  that  the  elicitor  may  want 
pertains  to  the  critical  features  behind  rules  and  procedures  that  the 
expert  is  using.  Hence,  the  expert  may  ask  what  made  the  expert  come  up 
with  a  particular  conclusion. 

In  terms  of  generalization,  the  elicitor  is  trying  to  find  out  the 
general  principles  and  rule  that  the  expert  is  using  while  forming  a 
solution  to  the  problem.  The  elicitor  may  ask  questions  like  "Is  that  a 
general  rule?".  This  type  of  question  extends  the  clarification- type 
question  of  finding  critical  features  by  seeing  what  other  instances  or 
class  of  instances  will  prompt  usage  of  a  particular  rule  or  problem 
solving  procedure. 

The  elicitor  may  also  challenge  the  expert  to  consider  alternative 
hypotheses  or  solutions.  Questions  like  "what  other  conclusions  could  be 
drawn  from  this?"  are  appropriate.  Often  an  expert  will  have  responded  to 
different  pieces  of  data  in  ways  which  suggest  different  (or  opposing) 
conclusions  are  possible.  If  the  elicitor  can  catch  these,  he  might  call 
these  apparent  contradictions  to  the  expert  and  ask  him  to  resolve  them. 
The  elicitor  (or  the  consultant)  should  be  prepared  to  play  "devil's 
advocate"  in  dealing  with  the  expert.  While  this  may  be  difficult  at 
first,  if  more  than  one  expert  is  used  in  the  same  problem,  a  list  of 
these  alternate  hypotheses  might  be  developed  across  experts.  The 
elicitor  can  draw  from  this  list  to  challenge  subsequent  experts. 
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5.0  GUIDELINES  FOR  USING  KNOWLEDGE  ELICITATION 
METHODOLOGIES  AND  TECHNIQUES 


In  Section  4,  we  outlined  two  contrasting  knowledge  elicitation 
methodologies,  interpretative  and  generative,  as  well  as  different 
elicitation  techniques  (e.g.,  problem  solving  interview)  for  these 
methodologies.  In  this  section,  we  offer  guidelines  to  help  a  prospective 
knowledge  engineer  decide  when  to  use  which  methodology  and  which 
techniques.  Such  a  decision  could  potentially  involve  several 
considerations:  characteristics  of  the  problem  area,  characteristics  of 
the  expert,  the  eventual  application  to  which  the  knowledge  is  to  be  used 
and  the  types  of  knowledge  the  elicitor  is  trying  to  obtain. 

5.1  Problem  Area  Characteristics 

Our  present  studies  addressed  the  first  two  factors  although  we  offer 
guidelines  that  take  into  account  all  four  factors.  The  problem 
characteristics  play  an  important  role  in  determining  which  methodology 
and  elicitation  techniques  to  use.  In  the  present  study,  two  problem 
areas  were  examined:  situation  development  and  order  of  battle  analysis. 
While  both  involve  intelligence  analysis,  the  two  problem  areas  have 
important  fundamental  differences. 

First,  situation  development  is  a  high-level  integration  task.  The 
situation  developer  must  integrate  diverse  bodies  of  information  that  can 
lead  to  a  rather  open-ended  set  of  conclusions  (there  are,  of  course,  some 
general  classes  of  conclusions,  e.g.,  attack,  defend,  etc.).  The  order  of 
battle  analyst,  on  the  other  hand,  operates  at  a  much  lower  level  of 
abstraction.  The  order  of  battle  analyst  works  with  a  limited  set  of 
templates  (e.g.,  motorized  rifle  division)  to  which  he  maps  incoming  data. 

The  difference  between  situation  development  and  order  of  battle 
analyses  can  be  characterized  by  several  features  which  have  more 
widespread  generality.  The  first  characteristic  is  the  level  of 
generality  that  the  problem  area  will  deal  with.  As  noted  above, 
situation  development  deals  with  information  at  a  much  more  general  level 
than  does  order  of  battle  analysis.  As  a  result,  we  argue  that  situation 
development  is  more  conducive  to  interview  techniques  than  order  of  battle 
analysis  since,  as  argued  in  Section  4,  interview  techniques  are  good  at 
eliciting  more  general  knowledge.  Because  order  of  battle  analysis 
primarily  deals  with  a  more  data-oriented  level  of  abstraction,  we  argue 
that  it  is  more  suitable  for  a  problem  solving  technique  than  situation 
development.  As  argued  in  Section  4,  problem  solving  techniques  are  best 
suited  for  eliciting  more  specific  or  data-oriented  knowledge. 

Another  critical  feature  which  distinguishes  situation  development 
from  order  of  battle  analysis  is  the  pool  of  solutions  that  could  possibly 
arise  out  of  the  analysis.  As  argued  above,  the  situation  developer  has  a 
wide  range  of  solutions  he  can  come  to  (i.e.,  the  enemy  can  have  numerous 
possible  courses  of  action  open  to  him)  whereas  the  order  of  battle 
analyst  is  working  with  a  more  limited  set  (i.e.,  there  are  only  so  many 
types  of  units  a  given  enemy  has) .  Having  a  large  set  of  potential 
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solutions,  particularly  when  many  of  them  might  be  novel,  requires  the 
expert  to  engage  in  a  more  generative  type  of  problem  solving  (i.e.,  the 
expert  must  "generate"  the  solution) .  This  suggests  that  a  form  of  the 
generative  methodology  might  be  most  suitable.  This  implies  that  for 
situation  development  experts,  the  generative  method  might  be  most 
suitable.  In  particular,  a  variant  of  the  data-driven  problem  solving 
approach  may  be  helpful  where  the  data  presented  are  at  a  higher  level  of 
generality  (e.g.,  a  developed  terrain  analysis,  a  complete  OB,  etc.). 

When  the  potential  solution  set  is  small  and  well-known  and  the 
problem  is  largely  deciding  which  solution  is  appropriate,  problem  solving 
will  largely  be  driven  by  the  pre-existing  solution  set  or  be  "top-down" 
in  nature.  This  is  descriptive  of  the  essence  of  order  of  battle 
analysis.  Hence,  the  interpretative  method  might  be  most  appropriate  for 
order  of  battle  analysts.  In  particular,  the  structured  interview 
technique  may  be  particularly  useful  in  eliciting  the  OB  analyst's  general 
problem  solving  approach,  while  the  top  down  problem  solving  technique  may 
be  useful  in  eliciting  the  frames  or  templates  the  expert  has  plus  the 
inference  rules  for  building  them  from  raw  data. 

Another  key  feature  which  characterizes  a  problem  solving  domain  (and 
was  alluded  to  in,  and  relates  to,  the  number  of  potential  solutions  that 
could  arise)  is  the  degree  of  novelty  associated  with  potential  events  and 
outcomes.  In  general,  novelty  requires  the  expert  to  act  in  a  generative 
mode,  constructing  new  knowledge  structures  and  new  solutions.  Such  a 
domain  feature  would  suggest  that  a  generative  elicitation  methodology, 
particularly  one  which  emphasizes  data  drive  problem  solving  might  be  most 
appropriate.  In  such  instances,  the  problem  should  be  adapted  to  include 
new  problems  or  new  data  configurations. 

Yet  another  key  feature  of  domain  is  the  degree  of  uncertainty 
associated  with  the  domain.  We  argue  that  resolving  uncertainty  requires 
the  execution  of  specific  procedures  and  evaluations  of  conclusions  they 
lead  to.  Kahneman  and  Tversky  (Kahneman,  Slovic  and  Tversky,  1982)  argue 
that  people  will  often  use  conceptual  (or  mental)  models  to  reason  about 
uncertainty.  Hence,  our  notion  of  mental  models,  coupled  with  inference 
rules,  may  play  a  critical  role  in  reasoning  about  uncertainty.  We 
suggest  that  problem  solving  techniques  are  most  useful  at  eliciting  these 
mental  models  and  inference  rules. 

One  feature  that  characterizes  domains  is  whether  the  tasks  the 
experts  perform  involve  synthesis  of  bits  of  data  (analogous  to  situation 
development)  or  pattern  matching  (e.g.,  order  of  battle  analysis).  This 
feature  may  be  correlated  with  the  number  of  potential  solutions  possible 
(since  the  presence  of  possible  solutions  typically  entails  an  emphasis  on 
synthesis  while  the  presence  of  few  potential  solutions  allows  for  greater 
pattern  matching) .  These  tasks  involve  generative  and  interpretative 
problem  solving,  respectively.  As  such  we  would  argue  that  data-driven 
problem  solving  and  top-down  problem  solving,  respectively,  would  be 
appropriate  knowledge  elicitation  techniques. 
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5.2 


Characteristics  of  the  Expert 

Characteristics  of  the  expert  from  whom  knowledge  Is  being  elicited 
are  also  an  important  consideration  In  deciding  which  knowledge 
elicitation  techniques  and  methodology  to  use.  Typically,  experts  with 
certain  characteristics  will  be  selected  for  particular  problem  areas. 
Therefore,  it  is  expected  that  correlation  between  types  of  expert 
characteristics  and  domain  features  will  exist.  However,  within  a  given 
problem  area,  there  are  bound  to  be  differences  across  experts,  some  of 
which  are  intrinsic  to  the  expert  himself  and  some  which  are  acquired  as  a 
result  of  experience. 

5.2.1  Intrinsic  qualities  of  the  expert.  Perhaps  the  most  important 

quality  an  expert  can  possess  (in  addition  to  his  expertise!)  is  the 
ability  to  articulate  his  expertise.  Indeed,  the  interview  techniques 
depend  entirely  on  this  ability.  If  experts  are  articulate,  then 
interview  techniques  are  appropriate  and  useful.  Also,  think  aloud 
problem  solving  will  be  valuable.  If  an  expert  is  not  articulate,  the 
elicitor  will  have  to  take  more  indirect  approaches  to  knowledge 
elicitation:  such  as  utilizing  problems  where  output  can  be  observed  and 

measured,  and  applying  the  more  experimental  techniques  such  as 
recall/reconstruction,  multidimensional  scaling  and  similarity  judgments, 
and  cloze  experiments. 

Another  dimension  which  characterizes  experts  is  their  cognitive 
style.  Some  experts  are  more  intuitive  while  others  are  more  analytical. 
Analytical  experts  may  follow  procedures  more  explicitly  and  utilize  data 
more.  As  such,  the  generative  method  may  be  more  appropriate  for 
analytically  inclined  experts  and  in  particular,  the  data-driven  problem 
solving  technique.  For  intuitive  experts,  the  interpretative  method  may 
be  more  appropriate  since  it  allows  the  intuitive  expert  to  deal  with 
generalities  and  not  get  caught  up  in  the  nitty-gritty  of  data.  Intuitive 
experts  may  be  good  at  analogical  reasoning  and  therefore  may  be 
particularly  good  at  high-level  similarity  judgments  as  well  as  recalling 
similar  episodes  from  memory  that  share  common  high-level  themes. 

5.2.2  Experimental  differences  across  experts.  There  are  two  broad 
categories  that  may  fall  under  the  heading  of  experimental  differences 
across  experts.  These  differences  acquired  from  training  (including 
real-world  experiences)  and  differences  as  a  result  of  an  assigned 
organizational  role.  With  regard  to  training,  two  broad  types  of  training 
occur.  The  first  is  a  classroom  and  book  training  which  typically 
features  doctrinal  approaches  to  problem  solving  and  a  general  discussion 
of  underlying  principles.  The  second  type  of  training  features  actual 
experience  (typically  in  real  world  contexts,  although  in  the  military 
arena  this  is  relatively  rare  with  realistic  problems. 

Classroom  or  book  knowledge  is  typically  conveyed  through  verbal 
media  (either  lectures  or  books).  Examples  tend  to  be  more  general  in 
nature,  hence  the  resulting  knowledge  representations  will  tend  to  be 
rather  abstract.  As  a  result  of  these  two  factors,  we  argue  that 
interview  techniques  are  appropriate  for  eliciting  "book"  knowledge. 
Experimental  knowledge  is  derived  from  doing  and  hence  may  be  more 
procedural  and  less  language-oriented  than  book  knowledge.  An  expert  may 
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not  be  as  easily  able  to  articulate  such  knowledge.  Hence,  problem 
solving  techniques,  particularly  those  rich  details,  may  be  most  useful  in 
eliciting  knowledge  derived  from  experience.  In  addition,  techniques  such 
as  critical  incidents  which  require  experts  to  draw  on  experiences  would 
also  be  useful. 

With  regard  to  organizational  role  differences  between  experts,  there 
are  a  number  of  distinguishing  factors.  The  first  factor  relates  to  the 
above  distinction  between  "book"  learning  and  experimental  learning,  and 
that  distinction  is  whether  the  expert  is  a  teacher  or  a  doer  (realizing 
of  course,  that  many  experts  are,  or  have  been,  both).  Teaching  is  the 
mirror  image  of  the  classroom  student.  Here,  the  teacher  must  be  able  to 
articulate  knowledge  and  understand  general  principles.  Hence,  the 
interview  techniques  may  be  especially  useful.  The  doer  is  similar  to  the 
student  involved  with  experimentially-based  training.  The  doer  focuses  on 
practical  applications  which  require  a  more  detailed  orientation  to 
knowledge,  as  well  as  knowing  the  procedures  and  their  nuances  for 
specific  situations.  Therefore,  a  problem  solving  approach  would  perhaps 
be  most  suitable  for  knowledge  elicitation  for  "doers."  Again,  techniques 
such  as  critical  incidents,  which  are  designed  to  capture  knowledge 
relating  to  experiences  would  also  be  suitable. 

Another  factor  which  distinguishes  different  experts  is  their 
position  within  the  organizational  hierarchy.  This  is  particularly  true 
in  the  military  setting  where  a  formal  ranking  system  is  established. 
Typically  within  hierarchical  organizations,  those  with  lower  tanks  focus 
more  on  circumscribed  tasks  which  are  detail  oriented.  Higher  ranking 
personnel  deal  more  with  the  "big  picture,"  deal  more  with  generalities 
and  conclusions,  and  tend  to  integrate  output  from  the  lower  levels.  This 
would  suggest,  broadly,  that  the  interpretative  method  might  be  more 
suitable  for  higher  ranking  experts  while  the  generative  method  might  be 
more  suitable  for  lower  ranking  experts.  There  is  an  important 
qualification  to  this.  The  data  driven  problem  solving  technique,  which 
is  part  of  the  generative  method,  is  good  at  capturing  the  integrative 
function.  A  modification  of  this  technique,  using  high  level  inputs, 
might  be  appropriate  for  high  level  experts.  The  decision  as  to  whether 
or  not  to  include  this  technique  might  be  driven  by  considerations  as  to 
whether  the  high  level  expert  has  a  well  formed  script  (such  as  METT-T)  to 
perform  the  integrative  function  or  whether  the  integration  has  to  be 
performed  in  a  "generative"  fashion,  i.e.,  without  a  predetermined  plan 
for  integrating. 

5 . 3  The  Application  Area  That  the  Elicited  Knowledge  is  to  Serve 

One  consideration  which  may  guide  knowledge  elicitation  is  how  the 
elicited  knowledge  will  be  put  to  use.  We  identify  three  applications 
which  could  benefit  from  knowledge  elicitation  and  building  models  of 
expert  knowledge.  These  are  building  expert  systems,  training,  and 
identification  of  experts  and  task  requirements. 

5.3.1  Expert  systems.  Knowledge  elicitation  can  benefit  the  design 
and  construction  of  expert  systems  by  providing  the  knowledge  base  the 
systems  used  and  the  algorithms  for  acting  on  that  knowledge  base  and 
producing  solutions  to  problems.  Two  things  characterize  expert  systems: 
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the  need  for  explicit  procedures  and  the  need  for  concrete  output.  Hence, 
problem  solving  techniques  are  probably  the  most  important  elicitation 
technique  to  use. 

We  have  discussed  two  techniques  which  utilize  problem  solving: 
top-down  problem  solving  and  data-driven  problem  solving.  Each  of  these 
techniques  can  serve  a  useful  role  in  expert  system  design.  Top-down 
problem  solving  can  be  useful  in  eliciting  the  general  problem  solving 
process  and  steps  that  experts  go  through.  Data  driven  problem  solving 
can  be  useful  at  eliciting  the  situation-specific  variations  in  the 
general  process,  as  well  as  how  experts  respond  to  specific  pieces  or 
groups  of  data. 

5.3.2  Training.  Training  differs  from  expert  system  design  in  that 
the  emphasis  in  training  is  typically  on  problem  solving  processes  rather 
than  output.  There  is  stronger  emphasis  on  the  rationales  that  students 
have  compared  to  say,  what  answer  they  come  up  with  (especially  in 
military  examples  where  there  may  not  always  be  a  "right"  answer) . 

Training  typically  involves  teaching  general  principles,  problem  solving 
procedures,  and  allowing  students  to  pick  up  the  details  through 
experience.  Hence,  the  interpretative  elicitation  method  may  be  more 
suitable  in  eliciting  how  experts  generally  solve  problems,  which  in  turn 
could  serve  as  the  model  for  teaching  students . 

However,  as  argued,  the  emphasis  is  also  on  the  rationale  underlying 
the  problem  solving  process  as  well  as  the  process  itself.  As  we  have 
argued,  these  rationales  can  be  represented  by  mental  models.  So  the 
elicitor  must  tailor  his  elicitation  techniques  to  get  at  these  models. 
Techniques  which  may  be  helpful  in  doing  this  include  interviews,  critical 
incidents,  and  those  which  employ  analogical  reasoning  such  as  sorting 
experiments . 

5.3.3  Identification  of  experts.  Often  it  is  necessary  to  identify 
experts  or  expert  requirements  for  particular  jobs,  like  consultation, 
etc.  Here  the  interest  is  in  identifying  what  capabilities  prospective 
experts  have  and  mapping  them  to  the  requirements  of  the  job  (or  whatever 
else  the  expertise  requires).  While  expertise  is  domain  specific,  and  the 
types  of  knowledge  that  experts  have  will  vary  from  domain  to  domain, 
nevertheless  there  are  still  general  differences  between  the  way  experts 
solve  problems  and  the  way  novices  solve  them.  Hence,  regardless  of  the 
domain,  we  would  expect  experts  to  exhibit  a  number  of  traits. 

First,  experts  will  have  a  great  deal  of  factual  knowledge  about 
their  domain.  Such  knowledge  might  be  captured  in  frames,  semantic  nets 
and  production  rules.  A  good  technique  for  eliciting  this  type  of 
knowledge  is  the  data-driven  problem  solving  technique  supplemented  by 
questions  designed  to  flesh  out  the  details  of  the  knowledge  being 
elicited  (e.g.,  what  are  the  components  of  a  motorize  rifle  division?). 
These  questions  should  be  generated  using  the  structured  interview 
technique  which  is  designed  to  flesh  out  the  different  slots  or  context 
components  of  knowledge  being  elicited. 

A  second  feature  that  characterizes  expertise  is  that  experts  have  an 
in-depth  causal  understanding  of  how  events  are  produced  in  their  domain 
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of  expertise  and  the  underlying  rationale  behind  their  problem  solving 
procedures  and  how  those  procedures  relate  to  the  events  being  produced. 
Cloze  experiments  are  useful  in  eliciting  the  knowledge  underlying  how 
events  are  produced.  A  variation  of  cloze  experiments,  where  the  unfilled 
slot  or  event  is  a  future  one,  is  useful  at  eliciting  knowledge  regarding 
how  plans  will  influence  future  events. 

A  third  feature  that  characterizes  expertise  is  that  experts  have  a 
lot  of  high-level  planning  strategies  and  heuristics  which  allow  them  to 
set  up  problem  solving,  and  generate  plans  and  solutions.  This  high-level 
knowledge  is  best  elicited  by  the  interpretative  elicitation  methodology, 
particularly  by  using  top-down  problem  solving  over  a  wide  range  of 
problems.  The  elicitor  can  take  this  one  step  further  to  understand  what 
criteria  experts  use  to  choose  among  high-level  planning  strategies  and 
mapping  them  to  specific  problems  by  conducting  sorting  and  similarity 
judgment  experiments  using  the  different  problems  and  solutions  of  the 
top-down  problem  solving  technique. 

A  fourth  feature  that  characterizes  expertise  is  that  experts  have 
well- integrated  knowledge  that  links  high-level  knowledge  to  low-level 
knowledge.  One  method  of  testing  for  this  would  be  to  apply  both  the 
interpretative  and  generative  methodologies  on  the  expert.  Two  findings 
should  come  out  of  this  if  the  subject  is  in  fact  an  expert.  First  both 
methods  should  be  successful  in  eliciting  a  lot  of  knowledge.  Second,  and 
most  important,  there  should  be  strong  linkages  between  the  knowledge 
driven  from  a  bottom-up  strategy  and  a  top-down  strategy  (i.e.,  if 
knowledge  is  well  integrated  then  using  a  top-down  strategy  should  not 
only  elicit  high-level  knowledge  but  low-level  as  well,  and  vice  versa. 
Experts  should  develop  a  similar  solution  and  reason  using  largely  the 
same  knowledge,  if  in  fact  high-level  and  low-level  knowledge  are  well 
integrated) . 

5.4  Types  of  Knowledge 

The  elicitor  may  be  interested  in  certain  types  of  knowledge  such  as 
inference,  decision  making,  planning,  etc.  These  different  types  of 
knowledge  may  influence  the  best  ways  to  go  about  eliciting  them.  A 
general  rule  in  elicitation  would  be  to  set  up  the  session  so  that  the 
expert  is  performing  those  tasks  that  involve  the  type  of  knowledge  the 
elicitor  is  interested  in.  Hence,  problem  solving  techniques  are  strongly 
indicated. 

If  the  desired  knowledge  is  inference -related,  the  data  driven 
technique  seems  most  suitable.  Decision  making  and  planning  may  be  best 
elicited  by  the  top-down  problem  solving  approach  since  those  tasks 
typically  occur  once  the  inference  mechanisms  have  been  run  and  the 
context  is  determined.  Knowledge  regarding  facts  and  data  (if  that  is  the 
desired  knowledge)  might  be  easily  elicited  from  interview  techniques, 
providing  of  course,  sufficient  indices  are  provided  to  the  expert.  Here, 
using  a  subject  matter  expert  as  a  consultant  may  be  useful  in  generating 
such  primes.  Managing  resources  both  human  and  others  may  be  another  type 
of  desired  knowledge.  Such  knowledge  integrates  planning,  decision  making 
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and  others  and  Is  generally  a  very  high  level  task.  As  such,  the 
interpretative  methodology  might  be  the  most  suitable  for  knowledge 
elicitation. 

5.5  General  Considerations 

There  are  a  number  of  general  issues  to  consider  in  knowledge 
elicitation  other  than  what  methodology  or  techniques  should  be  used  for 
what  domain  or  what  expert.  These  include  issues  like  how  many  experts 
should  be  used  and  who  they  should  be,  how  long  should  the  elicitation 
sessions  be,  and  how  to  evaluate  the  success  of  knowledge  elicitation  as 
it  is  ongoing,  including  the  questions  of  whether  and  when  to  change 
strategy. 

5.5.1  Finding  the  rlpht  (number  of)  experts.  Just  as  mining  for 
gold  depends  more  on  whether  or  not  there  is  gold  (than  on  what  tool  the 
miner  brings)  eliciting  expert  knowledge  is  only  as  successful  as  the 
availability  of  expert  knowledge  to  elicit.  Unfortunately,  not  all 
domains  have  recognized  experts  or  clear  cut  standards  to  measure 
expertise  (as  does  chess  or  baseball).  The  military  domain  is  even 
further  hindered  since  expertise  is  being  developed  under  peacetime 
conditions  which  will  be  dramatically  different  from  wartime  conditions . 

Still  there  are  three  criteria  that  can  be  used  to  select  experts 
from  the  pool  of  people  who  do  the  job  and  possess  knowledge  the  elicitor 
needs.  First,  the  rank  of  the  candidate  expert  gives  an  indication  of  the 
military  community's  opinion  of  the  candidate's  expertise.  Second,  years 
of  experience,  particularly  those  relating  to  the  actual  domain  of 
interest.  Certain  experiences  like  practical  training,  such  as  exercises 
and  assignments  to  locations  to  perform  domain- relevant  duties  (e.g.,  sent 
on  a  contingency  mission  to  South  America) ,  provide  important  indicators 
of  relative  expertise.  Finally,  peer  nomination  can  be  used,  where 
knowledgeable  people  in  the  domain  field  choose  the  people  whom  they 
believe  to  be  the  most  expert.  If  feasible,  a  fourth  criterion  could  be 
used,  i.e.,  using  knowledge  elicitation  techniques  to  screen  experts  as 
described  earlier. 

Once  a  pool  of  experts  has  been  identified  and  made  available,  the 
elicitor  must  decide  how  many  to  use.  The  elicitor  may  have  to  trade-off 
time  constraints  (both  his  own  and  availability  of  the  expert's  time) 
against  the  desire  to  be  as  complete  as  possible.  Using  multiple  experts 
can  have  two  benefits:  first,  to  supplement  each  other's  knowledge  and 
second,  to  corroborate  each  other's  knowledge. 

Ideally,  the  use  of  multiple  experts  would  primarily  be  to 
corroborate  knowledge  rather  than  to  supplement  it.  This  is  so  because 
using  several  experts  to  build  a  single  knowledge  representation  runs  the 
risk  of  developing  an  aggregated  knowledge  structure  which  perverts  the 
interconnections  between  the  separate  pieces  of  knowledge  (i.e.,  different 
experts  may  interconnect  knowledge  differently,  hence  to  "mix  and  match" 
across  experts  may  lead  to  an  erroneously  connected  knowledge  model). 
However,  practically  speaking,  it  is  typically  impossible  to  elicit  a 
complete  knowledge  base  from  a  singe  expert.  In  such  cases,  it  is 
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probably  most  desirable  to  elicit  knowledge  base  segments  In  depth  from 
single  experts  and  piece  these  together.  For  example,  In  eliciting 
knowledge  regarding  situation  development,  it  may  be  useful  to  elicit 
knowledge  regarding  terrain  analysis  from  one  expert,  order  of  battle  from 
another,  etc. 

It  is  desirable  to  replicate  the  elicitation  process  over  a  sample  of 
experts.  The  number  of  replications  depends  upon  the  degree  of  consensus 
the  experts  provide.  If  two  experts  show  nearly  identical  responses,  then 
this  may  suffice.  If  the  experts  are  markedly  different,  then  a  larger 
sample  may  be  required  (much  the  same  way  any  experimental  design  is 
generated- 'the  size  of  the  sample  required  depends  upon  subject 
variability) .  Additional  experts  serve  not  only  to  get  a  measure  of 
"central  tendency"  of  experts  but  also  to  explain  why  the  differences 
exist  (i.e.,  are  there  acceptable  variations  in  the  process?). 

5.5.2  Length  of  elicitation  sessions.  Scheduling  sessions  for 
elicitation  may  be  determined  more  by  availability  of  the  experts'  time 
than  anything  else.  However,  if  the  elicitor  has  some  flexibility,  then 
we  offer  some  guidelines.  In  general,  the  elicitor  must  trade  off  the 
need  to  give  the  expert  sufficient  time  to  get  "into"  the  elicitation  and 
meet  technique  requirements,  against  the  desire  to  avoid  fatiguing  the 
expert  and  the  desire  to  obtain  feedback  between  elicitation  sessions  to 
use  in  guiding  further  sessions. 

The  amount  of  time  required  for  techniques  will  depend  upon  the 
technique  itself.  In  general,  experimental  techniques  require  the  least 
amount  of  time  (e.g.,  cloze  experiments,  sorting  experiments,  etc.)  An 
hour  or  two  per  technique  will  typically  suffice.  Problem  solving 
techniques  may  be  the  longest  in  that  they  require  gaining  familiarity 
with  the  problem,  sifting  through  data,  etc.  Fortunately,  problems  are 
usually  set  up  beforehand  and  the  elicitor  may  not  need  to  benefit  much 
from  between- session  analysis  to  guide  future  sessions.  The  exceptions  to 
this  include  cases  where  the  elicitor  may  wish  to  chose  between  patterns 
of  data  he  will  present  to  the  expert,  or  generate  questions  about 
hypotheses  the  expert  has  been  developing. 

Interview  techniques  are  the  most  flexible  In  that,  theoretically, 
interviews  can  last  anywhere  from  a  few  seconds  to  years.  Since  interview 
techniques  depend  heavily  on  follow-up  questions  and  developing  trains  of 
thought,  these  would  benefit  most  by  being  able  to  break  between  sessions. 
These  breaks  can  be  used  to  evaluate  what  has  been  said  and  to  decide 
where  to  proceed. 

Based  on  this  discussion,  we  offer  the  following  general  guidelines: 
problem  solving  techniques  should  be  done  within  a  single  continuous 
session  (see  exceptions  noted  above),  so  that  the  problem  stays  fresh  in 
the  expert's  mind.  Future  problem  solving  sessions  can  be  devoted  to 
giving  the  expert  variations  on  the  problem  to  develop  generalizations  and 
situation-specific  variations  of  the  problem  solving  process.  Interview 
techniques  should  be  spaced  over  several  sessions  to  give  the  elicitor  a 
chance  to  analyze  what  has  transpired  and  develop  follow-up  lines  of 
questioning  to  use.  These  sessions  should  be  shorter,  the  length  of  time 
depending  upon  the  interval  in  between  sessions  (i.e.,  the  longer  the 
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interval  the  more  time  the  elicitor  has  to  analyze  the  previous  material) . 
However,  if  possible,  it  is  desirable  to  conduct  daily  elicitation 
sessions  so  that  the  materials  stay  fresh  in  the  expert's  mind. 

Therefore,  the  optimal  daily  time  for  interview  techniques  may  be  on  the 
order  of  two  hours. 

5.5.3  Ongoing  evaluations  of  the  knowledge  elicitation  process. 

This  is  a  very  important  but  very  difficult  task  for  the  elicitor  to 
perform.  While  we  have  offered  guidelines  on  which  techniques  and 
methodologies  to  use  on  whom,  this  does  not  guarantee  a  successful 
knowledge  elicitation  plan.  The  elicitor  may  have  to  evaluate  his 
strategy  "on-line"  and  make  decisions  as  to  whether  changes  need  to  be 
made.  Such  changes  may  be  relatively  minor  such  as  rephrasing  questions, 
or  may  be  drastic  if  it  seems  to  be  the  case  that  the  whole  strategy  is 
not  working. 

The  elicitor  has  three  potential  sources  of  feedback  regarding  the 
success  of  the  knowledge  elicitation  strategy:  his  own  perceptions, 
feedback  from  the  expert,  and  feedback  from  the  subject  matter  expert 
consultant  (if  one  is  being  used).  From  the  elicitor's  perspective,  he 
must  decide  whether  the  expert  is  giving  him  the  desired  knowledge  (or  at 
least  whether  it  seems  like  the  expert  is  giving  the  desired  knowledge) . 

If  the  expert  seems  to  be  responding  to  the  elicitor's  probes  or  is  giving 
the  class  of  answers  that  the  elicitor's  techniques  indicate  (e.g., 
similarity  judgments,  inferences,  etc.),  then  the  elicitation  is  probably 
going  well.  However,  if  the  expert  has  trouble  coming  up  with  these 
responses,  it  could  suggest  three  things,  the  expert  is  having  trouble 
understanding  the  elicitor,  the  wrong  techniques  are  being  used  to  capture 
what  the  expert  knows,  or  the  expert  may  really  not  have  the  appropriate 
knowledge  and  the  elicitation  may  need  to  be  abandoned. 

The  expert  himself  may  be  able  to  distinguish  among  these 
possibilities.  In  our  experience,  experts  will  often  openly  tell  an 
elicitor  when  he  doesn't  possess  the  knowledge  the  elicitor  is  trying  to 
obtain.  Also,  a  consultant  may  offer  an  opinion  on  the  matter  as  well. 

In  general  though,  the  elicitor  should  follow  a  four  step  process  for 
altering  the  elicitation  process. 

The  first  step  is  rephrase  the  questions  to  clarify  them  for  the 
expert.  This  may  include  giving  concrete  examples  (e.g.,  instead  of 
asking  "What  is  a  motorized  rifle  division  made  up  of?",  the  question 
would  be  rephrased,  "Are  there  any  armored  units  in  a  motorized  rifle 
division?").  Another  approach  would  be  to  tell  the  expert  about  a 
previous  expert's  response  to  the  same  question  (or  a  doctrinal  one- -a 
consultant  may  help  generate  this)  and  allow  the  expert  to  agree, 
disagree,  or  comment  on  the  response.  The  elicitor  tries  to  find  a  format 
that  the  expert  is  comfortable  with  and  adopts  the  strategy  of  phrasing 
questions  in  that  way. 

If  the  elicitor  cannot  find  a  way  to  phrase  questions  that  the  expert 
is  comfortable  with,  the  elicitor  may  need  to  switch  techniques  with  the 
expert.  Using  our  guidelines  or  direct  feedback  from  the  expert  may  given 
an  indication  as  to  what  techniques  might  be  useful  (for  this  reason  the 
elicitor  should  have  the  capability  to  do  more  than  one  technique  with 
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each  expert).  If  the  new  technique  does  not  seem  to  work,  the  elicitor 
should  switch  back  to  the  tactic  of  rephrasing  questions  before  switching 
techniques  again.  Of  course,  if  this  cycle  gets  repeated  too  often,  the 
elicitor  should  proceed  to  his  next  line  of  recourse. 

If  the  elicitor  is  having  trouble  finding  a  technique  that  the  expert 
can  respond  to,  it  may  suggest  that  the  expert  does  not  have  the 
appropriate  expertise  for  the  elicitor' s  purpose.  If  this  is  the  case, 
the  elicitor  may  wish  to  change  strategies  and  ask  the  expert  to  speak 
about  his  area  of  expertise.  Here  the  elicitor  can  ask  the  expert  what  he 
would  feel  comfortable  talking  about  and  engage  the  expert  in  a  structured 
interview  (which  requires  little  preparation  and  can  be  done  with  little 
or  no  expertise  on  the  elicitor' s  part). 

If  the  elicitor  finds  that  the  expert's  area  of  expertise  is  not 
relevant  to  the  domain  he  is  targeting  (the  consultant  may  give  input  to 
this),  then  the  elicitor  may  decide  to  end  the  session  completely. 
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6.0  EXPERIMENTAL/RESEARCH  METHOD 


6.1  Preliminary  Research  and  Pilot  Studies. 

It  was  hypothesized  that  the  methods  of  elicitation  likely  to  be  most 
effective  in  capturing  expert  problem  solving  knowledge  are  ones  that  are 
compatible  with  the  way  the  domain  expert's  knowledge  is  structured  which 
is,  in  turn,  influenced  by  the  characteristics  of  his  problem  domain.  In 
order  to  develop  methods  of  knowledge  elicitation  compatible  with  our 
target  domain,  some  of  our  early  efforts  focused  on  learning  more  about 
the  military  intelligence  domain  that  would  serve  as  our  research  testbed. 
To  accomplish  this,  a  series  of  pilot  elicitation  studies  were  conducted 
in  the  context  of  the  selected  problem  domain  of  tactical  intelligence. 
These  efforts  were  largely  open-ended  and  exploratory,  for  the  purpose  of 
better  understanding  the  characteristics  and  constraints  presented  by  this 
problem  domain,  that  would  have  to  be  factored  into  the  design  of 
alternative  methods  of  elicitation  that  we  would  later  implement  and  study 
at  greater  length. 

One  of  the  pilot  efforts  involved  a  rather  brief  and  preliminary 
elicitation  case  study  in  which  a  consultant  from  Burdeshaw  Associates 
Ltd.  (BAL)  participated  as  the  expert  and  several  members  of  the  technical 
staff  of  DSC  and  some  of  our  consultants  from  Smart  Systems  Technology 
(SST)  jointly  participated  as  elicitors.  Some  preliminary  investigations 
of  elicitation  methods  in  the  domain  of  combat  intelligence  was  also 
conducted  at  Fort  Leavenworth,  Kansas  and  at  Fort  Huachuca,  Arizona. 

6.1.1  Pilot  elicitation  session  at  DSC.  A  principal  objective  of 
the  pilot  elicitation  session  conducted  at  DSC  was  to  exercise  and 
rehearse  the  kind  of  elicitation  material  that  might  be  used  in  later 
investigations,  by  thinking  (and  working)  through  a  concrete  example. 

This  would  not  only  help  "shake  down"  our  initial  ideas  for  prospective 
alternative  methodologies ,  but  would  serve  as  something  of  an  introduction 
to  the  Army  intelligence  domain.  A  range  of  possible  elicitation 
perspectives  were  considered,  varying  in  terms  of  the  kind  of  knowledge 
tapped  (e.g.,  elicitation  of  decision  making  processes  vs.  elicitation  of 
subjective  assessment  processes  vs.  elicitation  of  standard  information 
about  objects  of  the  domain),  the  source  or  basis  of  expertise  (e.g., 
learning  from  sourcebooks  vs.  first-hand  experience),  and  the  types  of 
elicitation  (e.g.,  structured  interview  vs.  protocol  analysis).  These 
alternative  perspectives  also  reflected  a  range  of  potential  research 
foci. 


Although  a  range  of  possible  case  settings  was  also  considered,  the 
case  materials  that  were  ultimately  used  were  those  already  on  hand  at 
DSC,  originally  developed  for  research  done  at  Fort  Leavenworth  on 
decision  making  under  "time  stress"  in  another  ARI -sponsored  Army  project. 
The  case  concerned  a  fairly  standard  NATO  scenario  in  which  The  Warsaw 
Pact  has  invaded  West  Germany.  mis  Whitehead  (MG,  retired),  our 
consultant  from  BAL,  played  the  role  of  "expert";  in  this  case,  an 
experienced  G2  working  at  the  division  level. 
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In  the  first  phase  of  this  pilot  elicitation  session,  General 
Whitehead  was  asked  to  address  his  remarks  to  a  "novice"  G2  who  is 
preparing  to  take  up  the  post  of  divisional  G2  in  the  context  of  the 
scenario.  He  was  further  asked  that  the  knowledge  that  he  would  impart 
should  be  limited  to  that  which  is  not  available  in  existing  publications, 
nor  routinely  learned  in  junior  level  officer  training  courses.  This 
could  include  guidance  that  is  scenario  specific,  knowledge  that  is  too 
new  or  controversial  to  be  embodied  in  existing  documents,  and  knowledge 
that  is  organizationally  sensitive  or  goes  beyond  general  doctrine.  The 
session  was  further  focused  on  intelligence  collection  management  within 
the  G2  staff,  and  on  knowledge  that  can  be  used  to  guide  actions  in  this 
area  (as  contrasted  with  description  or  causal  generalization) . 

The  pilot  elicitation  session  with  General  Whitehead  was  divided  into 
three  segments.  In  the  first  part  of  the  session.  General  Whitehead  was 
given  the  background  to  the  problem  scenario  and  asked  to  respond  at  a 
rather  broad  level,  with  general  wisdom  for  the  performance  of  data 
collection.  In  the  second  part  of  the  pilot  session,  our  expert  was  asked 
to  respond  to  specific  pieces  of  information  provided  in  the  unfolding 
problem  scenario.  In  the  third  segment  of  this  pilot  study,  a  top-down 
interview  format  was  adopted. 

Pilot  Study  Findings .  This  pilot  effort  was  analyzed  by  DSC  from  two 
different  perspectives.  An  examination  from  a  decision  analytic 
perspective,  by  Dr.  Rex  Brown  of  DSC,  is  presented  in  Appendix  A,  along 
with  detailed  records  of  this  pilot  session  and  commentary  contributed  by 
some  other  participants.  Observations  from  a  cognitive  processes 
perspective  with  implications  for  our  subsequent  elicitation  methodology 
are  presented  below. 

In  the  first  part  of  the  session  when  our  expert  was  asked  to  provide 
very  general  guidance  for  intelligence  collection,  it  was  found  that  while 
he  was  able  to  generate  a  few  broad  guidelines  in  response  to  this 
request,  he  relied  extensively  on  the  use  of  specific  features  of  the 
scenario  to  cue  his  generalized  advice.  After  providing  some  broad  and 
rather  abstract  form  of  guidance,  he  would  typically  elaborate  on  and 
explain  the  generalization  in  terms  of  specific  examples.  It  appeared 
that  his  lower  level  or  generative  knowledge  was  very  well  integrated  with 
(i.e.,  closely  connected  to  corresponding)  higher  level  or  interpretative 
knowledge.  Moreover,  since  he  was  less  experienced  as  a  lecturer  than  as 
an  implementer  of  these  general  principles,  it  was  difficult  for  him  to 
access  all  the  relevant  abstract  and  general  concepts  without  frequent 
reference  to  the  more  concrete  objects  and  operable  processes  to  which 
they  are  tied,  and  with  which  they  would  be  used  in  actual  problem 
solving.  Thus,  the  specific  problem  materials  appeared  to  provide  a 
useful  vehicle  for  retrieval  from  memory,  as  well  as  for  an  explanation  of 
the  higher  level  knowledge  that  was  sought  in  this  part  of  our  pilot 
session. 

In  the  second  part  of  the  pilot  session,  our  expert  was  asked  to  use 
the  specific  case  problem  and  "play"  out  the  scenario,  both  making 
decisions  based  on  the  information  available  and  imagining  events 
unfolding.  By  asking  our  expert  to  make  inferences  and  take  actions  based 
on  the  scenario,  we  expected  to  elicit  specific  rules  of  inference  and 
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action.  On  the  other  hand,  by  asking  him  to  extend  and  elaborate  on  the 
scenario  we  provided,  we  hoped  to  elicit  the  higher  level  knowledge  that 
provides  the  basis  for  expectations  and  assumptions.  While  some  concrete 
facts  and  rules  were  obtained  in  this  format  for  elicitation  from  General 
Whitehead,  most  of  his  remarks  addressed  the  general  goals  of  the  problem 
solving  and  corresponding  specific  goals  for  this  problem,  and  his  remarks 
generalized  to  more  specific  intelligence  collection  strategies.  One  of 
the  DSC  technical  staff  who  had  used  the  same  problem  scenario  with 
Lieutenant  Colonels  at  Fort  Leavenworth  commented  that  General  Whitehead's 
approach  was  very  different  from  that  of  the  Leavenworth  people:  the 
Leavenworth  subjects  focused  almost  exclusively  on  the  "micro  details"  of 
the  problem  while  General  Whitehead  appeared  to  focus  on  general  strategic 
issues.  This  difference  in  approach  supported  the  notion  that  despite  the 
intention  of  the  elicitor  to  tap  specific  and  concrete  versus  general  and 
abstract  knowledge  with  a  particular  method  of  elicitation,  a  subject's 
responses  may  be  greatly  influenced  a  priori  by  the  level  of  abstraction 
at  which  he  is  accustomed  to  working  the  problem.  If  this  is  the  case, 
one  would  expect  to  find  that  higher  level  officers  who  occupy  the  command 
or  "top  management”  positions  within  a  given  functional  area  will  respond 
to  most  methods  of  elicitation  with  knowledge  cast  in  more  general  and 
abstract  terms  (or  closely  linked  to  knowledge  of  this  fora)  while 
individuals  who  work  at  more  detailed  levels  of  that  function  will  respond 
to  elicitation  with  problem-solving  knowledge  that  is  largely  procedural 
and  driven  by  specific  and  concrete  information. 

The  third  part  of  this  pilot  session  followed  an  interactive 
interview  format.  A  different  member  of  the  team  took  the  role  of 
elicitor  and  began  to  ask  General  Whitehead  to  describe  why  he  would 
recommend  or  follow  the  intelligence  collection  strategies  he  described 
earlier.  This  prompted  General  Whitehead  to  draw  a  more  direct  link 
between  his  overall  problem  goals  and  these  broad  strategies  and  how  the 
broad  strategies  translated  into  more  specific  problem  solving  plans 
(i.e.,  by  the  nature  of  the  objects  involved).  The  elicitor  asked  for 
elaboration  on  these  responses  and  our  expert  provided  object  and  process 
descriptions.  In  subsequent  questions  in  which  the  elicitor  would  ask  for 
rather  general  information  (e.g.,  what  are  the  greatest  threats  to  you; 
tell  us  more  about  factors  contributing  to  the  success  of  the  attack...), 
our  expert  would  respond  first  with  a  description  of  specific  examples, 
generally  taken  from  the  context  of  the  case  problem,  and  would  then 
abstract  the  general  principles.  This  interactive  approach  to  elicitation 
of  higher  level  knowledge  appeared  to  be  fairly  effective.  However,  it 
was  also  noted  that  if  the  elicitor  is  trying  to  get  at  a  particular  level 
of  abstraction,  and  it  is  not  a  level  which  is  readily  accessed  by  direct 
questions  (or  not  in  a  form  that  corresponds  to  the  expert's  internal 
representation)  in  an  interview,  the  elicitor  will  have  to  be  persistent 
in  asking  for  that  information.  The  elicitor  may  need  to  rephrase  the 
question  in  different  ways  to  provide  alternative  means  of  cueing  the 
retrieval  or  reconstruction  of  the  desired  information,  and/or  allow  the 
expert  time  to  transform  information  from  the  problem  level  at  which  he  is 
more  aware  of  this  processing  to  the  level  of  interest  to  the  elicitor,  at 
which  the  expert  may  normally  be  less  conscious  of  his  problem  solving. 

6.1.2  Pilot  elicitation  studies  at  Fort  Huachuca.  In  addition  to 
the  elicitation  session  just  described,  pilot  elicitation  studies  were 
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conducted  at  Fort  Huachuca  and  Fort  Leavenworth.  This  effort  involved  two 
members  of  DSC's  project  team  and  a  consultant  from  BAL  who  served  as  an 
informal  liaison  as  well  as  guide  and  interpreter  of  the  military 
intelligence  domain. 

At  Fort  Huachuca,  two  pilot  elicitation  sessions  were  run  and 
different  elicitation  strategies  were  tested  in  each.  In  one  study,  a 
member  of  the  DSC  project  team  conducted  an  interview  with  a  Lieutenant 
Colonel  specializing  in  Army  intelligence.  The  interview  focused  on 
intelligence  gathering,  but  the  interview  format  was  open-ended  and 
unstructured  in  order  to  study  the  effectiveness  of  this  approach  to 
elicitation.  It  was  found  that  although  this  method  yielded  some  data 
that  could  be  encoded  into  interpretative  or  generative  knowledge 
components,  the  interview  was  generally  too  undirected  to  provide  enough 
information  to  yield  a  complete  representation  of  either  the 
interpretative  or  generative  knowledge  that  was  partially  tapped.  From 
the  experience  of  this  session,  it  was  concluded  that  the  elicitor  must 
play  a  more  active  role  in  directing  the  discussion  and  that  his 
questioning  should  be  done  with  the  objective  of  obtaining  more  complete 
representation;  in  particular,  one  that  includes  a  more  thorough 
description  and  explanation  of  the  problem  concepts  and  actions  being 
discussed. 

A  second  pilot  study  at  Fort  Huachuca  concerned  knowledge  elicitation 
sessions  in  which  the  intelligence  experts  (a  retired  division  G-2  and  a 
retired  corps  G-2)  worked  through  a  standard  Fulda  Gap  training  scenario. 
Each  subject  was  asked  to  play  the  intelligence  staff  role  that  they  were 
most  experienced  with,  and  to  describe  the  intelligence  information  they 
felt  would  be  necessary  to  have  in  order  to  do  their  job.  The  session  was 
conducted  with  relatively  little  interference  on  the  part  of  the  knowledge 
elicitors,  who  generally  restricted  themselves  to  questions  to  clarify  the 
statements  made  by  the  experts.  The  problem  they  were  given  was  such  that 
the  experts  were  able  to  simulate  (describe)  the  intelligence  gathering 
process  as  well  as  the  information  they  would  typically  receive  as  a 
result  of  that  process.  This  approach  to  elicitation  was  found  to  provide 
both  a  greater  depth  of  information  (in  the  context  of  the  specific 
problem)  and  a  wider  range  of  problem- relevant  Information  than  what  was 
obtained  in  the  open-ended  interview.  The  latter  finding  may  have  been 
the  result  of  providing,  through  the  use  of  the  specific  problem,  a  much 
greater  and  more  varied  set  of  cues  for  retrieval  and  reconstruction  of 
relevant  knowledge  from  the  expert’s  memory.  More  generally,  these 
results  suggest  that  when  working  on  specific  problems,  the  expert  may 
call  in  a  wider  range  of  knowledge  structures  than  when  he  attempts  to 
describe  or  discuss  his  problem  area  without  a  particular  focus  or 
well-defined  scope.  An  implication  of  this  finding,  regarding  the  use  of 
specific  problem  solving  for  elicitation,  is  that  the  knowledge  elicitor 
may  wish  to  define  his  problem  scope  more  narrowly  until  he  is  ready  to 
tackle  the  wider  range  and  extent  of  the  knowledge  sources  and  structures 
that  may  be  evoked. 

6.1.3  Pilot  elicitation  studies  at  Fort  Leavenworth.  At  Fort 
Leavenworth,  we  investigated  the  usefulness  of  observing  group  decision 
making  as  an  elicitation  technique.  This  evaluation  was  based  on  two 
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group  decision  making  exercises  observed  at  the  Command  and  General  Staff 
College  (CGSC) . 

The  decision  making  took  place  in  a  classroom  exercise  in  which  the 
instructor  played  the  role  of  division  commander  and  the  students,  divided 
into  two  groups,  each  assumed  the  role  of  some  member  of  the  commander's 
G2  or  G3  staff.  The  students  were  briefed  on  the  division's  mission  and 
the  status  of  the  battlefield  at  the  point  when  they  took  up  their 
position  on  the  staff.  The  maps  displayed  on  the  walls  reflected  the 
battle  scenario's  current  status.  A  computer  terminal  in  the  classroom 
then  printed  out  a  series  of  brief  messages  similar  to  the  kind  of  reports 
that  a  division  commander's  staff  might  receive  in  the  course  of  tactical 
battlefield  operations.  The  sequence  of  messages  included  MOVREPs , 
SPOTREPs,  SITREPs,  weather  reports,  AIR  DEFWARNs,  friendly  equipment 
status,  friendly  personnel  status  and  other  reports  concerning  enemy 
locations  and  actions.  From  this  stream  of  data  the  student/staff  members 
pieced  together  a  picture  of  the  battlefield  and  recommended  friendly 
courses  of  action.  The  instructor/division  commander  would  periodically 
intervene  to  critique  the  estimates  and  plans  that  were  being  developed. 

In  this  group  exercise  three  sources  of  data  were  available:  direct 
observation  of  the  actions  of  the  participants,  listening  to  the 
discussions  they  had  between  them  and  copies  of  the  briefings  they  gave 
the  instructor  of  the  class.  Unfortunately,  there  was  very  little 
opportunity  to  direct  questions  at  the  participants,  either  to  clarify 
what  they  were  doing  or  to  attempt  to  build  knowledge  representations  of 
what  they  were  doing. 

As  a  result,  our  overall  evaluation  of  using  this  kind  of  exercise  as 
a  knowledge  elicitation  technique  as  an  initial  method  for  an  elicitor 
unfamiliar  with  the  problem  domain,  or  as  the  sole  method  on  which 
knowledge  representations  would  be  built,  was  rather  negative.  Two  major 
problems  with  this  approach  were:  (1)  trying  to  infer  how  the 
participants  were  arriving  at  their  decisions  and  (2)  trying  to  keep  track 
of  each  participant's  input  into  the  problem-solving  process,  since  there 
was  a  great  deal  of  interaction  among  the  individual  participants,  who 
were  typically  playing  more  than  one  role  in  this  staff  exercise.  Having 
a  domain  expert  present  to  explain  the  decision  making  process  was  very 
helpful.  He  was  only  able  to  explain  what  was  going  on  at  a  rather 
general  level,  however,  since  it  was  difficult  to  keep  up  with  the 
numerous  and  fairly  complex  interactions  that  were  going  on  in  parallel. 

If  it  were  a  matter  of  having  our  expert  consultant  track  and  explain  the 
observed  behavior  of  an  individual  domain  problem- solver,  this  technique 
may  have  been  far  more  effective  as  an  elicitation  approach  to  be  used 
rather  early  on  in  building  a  domain  knowledge  base. 

Since  the  knowledge  of  any  individual  member  of  a  division  G-2  staff 
must  include  a  thorough  understanding  of  how  his  problem-solving 
interfaces  with  that  of  others  in  the  G2  and  G3  functions,  observation  of 
problem-solving  in  this  realistic  group  setting  would  be  needed  at  some 
stage,  however.  This  could  take  place  perhaps  after  elicitation  of  the 
autonomous  problem-solving  knowledge  is  largely  complete  and  the  elicitor 
is  better  versed  in  the  domain.  At  that  point,  observation  of  the  group 
decision-making  process  should  enable  the  elicitor  to  capture  the 


knowledge  of  how  and  when  interaction  with  other  specialized  areas  or 
experts  should  occur  in  a  given  subdomain's  problem-solving. 

6.1.4  Implications  of  the  pilot  studies  for  subsequent  research. 

The  pilot  studies  provided  evidence  of  the  extensive  use  of  both 
interpretative  knowledge  (e.g.,  rather  abstract  goals  and  plans)  and 
generative  knowledge  (e.g.,  concrete  data  driven  actions  and  inferences) 
in  problem-solving  in  the  tactical  intelligence  domain.  Different  forms 
of  knowledge  also  appeared  to  be  used  at  different  levels  of  abstraction 
(e.g.,  object  versus  process  knowledge).  This  observation  supports  the 
hypothesized  need  to  develop  knowledge  representations  to  capture  both 
higher  levels  and  lower  levels  of  abstraction  and  corresponding  methods  of 
elicitation  for  acquisition  of  tactical  intelligence  expertise. 

The  elicitation  experiences  of  the  pilot  studies  further  indicated 
that  highly  unstructured  and  open-ended  elicitation  techniques  would  not 
be  as  effective  with  most  experts  in  tactical  intelligence  as  more 
directed  methods  would  be,  even  in  acquisition  of  high  level, 
interpretative  knowledge.  This  appears  to  be  due,  at  least  in  part,  to 
the  manner  in  which  these  experts  normally  function  in  their  domain. 

Higher  level  goals  such  as  supporting  the  commander's  accomplishment  of 
his  mission  remain  fairly  constant  across  problems,  while  the  actual  use 
of  intelligence  collection  and  analysis  strategies,  and  sequences  of 
actions  to  handle  a  particular  problem  situation,  depend  very  much  on  the 
specifics  of  a  tactical  situation.  Moreover,  since  most  of  these  experts 
are  not  typically  called  upon  to  explain  or  teach  their  job  to  absolute 
novices,  they  don't  have  a  fairly  complete  and  coherent  presentation  of 
problem-solving  in  their  specialty  prepared  for  the  asking  as  might  be  the 
case  with  experts  who  also  function  as  lecturers  and  instructors  in  their 
specialty. 

It  also  appeared  that,  based  on  the  Fort  Leavenworth  pilot  study, 
"naturalistic"  observation  might  not  be  a  suitable  method,  at  least  at  the 
initial  phases  of  knowledge  elicitation.  Although  the  experts  are  engaged 
in  solving  a  specific  problem  and  the  format  is  highly  structured  from 
their  perspective,  mission  intelligence  experts  realistically  work  as  part 
of  a  group  of  interacting  problem  solvers  operating  roughly  in  parallel. 
This  situation  is  generally  too  complex  and  unbounded  for  an  observer  who 
is  not  very  familiar  with  the  domain  to  make  effective  use  in  knowledge 
elicitation. 

For  the  initial  phases  of  elicitation,  and  in  our  preliminary 
research  on  methods,  it  appeared  to  be  wise  to  restrict  ourselves  to 
methods  in  which  only  one  expert's  knowledge  was  being  elicited  at  any  one 
time,  and  methods  that  provided  varying  degrees  of  structure  and  direction 
for  the  discussions  with  experts. 

Our  pilot  studies  also  emphasized  the  value  of  having  someone  on  the 
elicitation  team  with  domain  familiarity  present  at  the  elicitation 
sessions,  to  clarify  and  explain  to  us  what  the  military  personnel  were 
doing  and  saying,  and  similarly  translate  for  them  what  we  wanted  to  do  or 
needed  to  know.  The  effectiveness  of  the  elicitation  efforts  appeared  to 
be  so  much  enhanced  by  this  that  we  planned  to  have  a  team  member  with 
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military  experience  accompany  us  in  all  our  subsequent  elicitation 
sessions  with  military  intelligence  experts. 


Subsequent  to  the  theoretical  work  on  development  of  integrated 
representations  of  interpretative  versus  generative  knowledge,  and  of  some 
rather  general  elicitation  strategies  to  capture  either  interpretative  or 
generative  knowledge,  the  focus  of  the  project  shifted  to  preparation  for 
our  elicitation  field  studies.  This  involved  the  development  of 
experimental  methods  and  materials  that  would  satisfy  our  research  goals 
as  well  as  the  practical  constraints  imposed  by  the  military  environment 
in  which  our  studies  would  be  conducted. 

This  section  of  the  report  describes  the  subdomains  of  tactical 
intelligence  to  which  our  elicitation  methods  would  be  applied,  the 
environment  in  which  elicitations  were  conducted,  and  the  consequent 
design  of  experimental  materials  to  accommodate  the  characteristics  of 
this  particular  research  "testbed"  while  meeting  our  theory-based  research 
objectives.  The  procedures  actually  used  in  interpretative  versus 
generative  elicitations  are  subsequently  described,  followed  by  a  log  of 
the  actual  data  collection.  The  log  of  elicitations  includes  a 
description  of  the  interview  sessions  conducted  at  various  Army  bases 
including  the  time  and  location,  the  participants,  and  the  method  of 
elicitation  used. 

6 . 2  Field  Unit  Studies 

6.2.1  Domains  of  expertise  selected.  The  purpose  of  the  field  unit 
studies  was  to  test  the  two  elicitation  methodologies  (interpretative  and 
generative) .  Two  contrasting  domains  of  expertise  were  picked  as  testbeds 
and  the  efficacy  of  the  interpretative  and  generative  methods  were 
subsequently  compared.  In  selecting  the  two  domains,  we  were  driven  by 
two  considerations:  first,  that  the  domains  fall  within  the  category  of 
intelligence  analysis,  and  second,  that  they  represent  different  functions 
which  can  be  characterized  by  different  cognitive  processes.  The  two 
domains  chosen  were  situation  development  and  order  of  battle  analysis. 

Order  of  battle  analysts  produce  an  assessment  of  the  identification, 
strength,  command  structure,  and  disposition  of  personnel,  units  and 
equipment  of  the  enemy.  They  rely  heavily  on  templates  which  describe  the 
organization  and  composition  of  enemy  combat  forces.  Hence,  much  of  the 
order  of  battle  analysis  revolves  around  matching  data  to  templates,  which 
suggests  an  interpretative  approach  to  problem  solving. 

Situation  development  analysts  produce  a  broader  intelligence 
estimate  that  includes  the  order  of  battle  analysis,  terrain  and  weather 
information,  enemy  doctrine,  etc.  These  various  kinds  of  information  are 
combined  to  provide  the  commander  with  an  analysis  of  the  area  of 
operations  and  enemy  capabilities  that  can  influence  the  friendly  mission, 
thus  providing  a  basis  for  planning  friendly  operations.  Since  the 
primary  purpose  of  situation  development  is  to  infer  probable  enemy 
courses  of  action  by  aggregating  bodies  of  data,  one  would  expect 
situation  development  to  be  a  largely  generative  process. 
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6.2.2  Materials .  A  set  of  materials  was  developed  for  each  of  the 
techniques  used  in  the  field  studies.  For  the  structured  interview,  a  set 
of  templates  was  developed  for  the  difference  knowledge  structures .  As 
discussed  in  Section  4,  these  templates  are  used  to  generate  questions 
designed  to  fill  the  slots  in  the  structure.  These  templates  were  also 
used  in  the  data  analysis  (see  Section  7);  a  copy  of  these  templates  are 
presented  in  Appendix  B. 

For  the  problem  solving  technique ,  a  standard  TRADOC  teaching 
scenario  was  used.  The  scenario  depicted  a  U.S.  division  about  to  conduct 
a  passage  of  lines  through  a  division  in  a  defensive  posture  in  order  to 
go  on  the  offensive  against  Soviet  forces  in  West  Germany.  In  addition  to 
general  background  information,  data  was  available  in  the  form  of  a  Corps 
periodic  intelligence  report  (PERINTREP)  and  an  intelligence  summary 
(INTSUM)  from  the  division  currently  on  the  defensive.  Enemy  order  of 
battle  templates  were  also  provided.  For  the  data-driven  problem  solving 
technique,  the  INTSUM  and  PERINTREP  were  cut  up  into  individual  messages 
and  presented  one  at  a  time.  A  copy  of  the  problem  solving  scenario  is 
presented  in  Appendix  C. 

In  addition,  supplemental  techniques  of  critical  incidents  and 
similar  incidents  were  used.  Question  templates  for  these  techniques  are 
included  in  Appendix  B. 

6.2.3  Field  units  visited.  In  cooperation  with  the  Army  Research 
Institute  (ARI)  and  the  United  States  Army  Forces  Command  (FORSCOM),  Forts 
Bragg,  Carson  and  Huachuca  were  selected  as  sites  for  field  studies. 
Participating  in  the  studies  were  representatives  from  the  G-2  shops  of 
the  82nd  Airborne  Division  at  Ft.  Bragg;  the  4th  Infantry  Division  at 

Ft.  Carson;  and  instructors  from  the  U.S.  Army  Intelligence  Center  and 
School  (USAICS)  at  Ft.  Huachuca. 

6.2. 3.1  Ft.  Bragg.  Six  subjects  were  recruited  from  Ft.  Bragg: 

3  representing  order  of  battle  analysis  expertise;  3  representing 
situation  development  expertise.  Subjects  ranged  in  rank  from  Warrant 
Officer  to  Lieutenant  Colonel.  Subjects  had  a  variety  of  roles  within  the 
division  G-2  shop  including  order  of  battle  technician,  assistant  G-2,  and 
the  G-2  himself.  Each  subject  was  available  for  one -half  day  for 
knowledge  elicitation. 

6. 2. 3. 2  Ft.  Carson.  Five  subjects  were  recruited  from  Ft.  Bragg: 

3  representing  situation  development  expertise;  2  representing  order  of 
battle  expertise.  Subjects  ranged  in  rank  from  Staff  Sergeant  to  Major. 
Included  in  these  were  an  order  of  battle  analyst,  the  division  OB 
technician,  the  head  of  the  All  Source  Production  Secion  (ASPS),  the  head 
of  Collection  Management  and  Dissemination  section  (CM&D) ,  and  an 
assistant  G-2.  Each  subject  was  available  for  a  full  day  for  knowledge 
elicitation. 

6. 2. 3. 3  Ft.  Huachuca.  Nine  subjects  were  recruited  from 

Ft.  Huachuca:  5  representing  order  of  battle  expertise,  4  representing 
situation  development  expertise.  Subjects  ranges  in  rank  from  Sergeant  to 


Major.  All  subjects  were  Instructors  at  USAICS.  Subject  availability 
ranged  from  2-10  hours.  Subjects  were  run  in  blocks  of  2-3  hours  each  per 
day. 


6.2.4  Procedures .  A  total  of  20  subjects  were  used  in  field 
studies;  10  representing  situation  development  expertise  and  10 
representing  order  of  battle  expertise.  Within  each  functional 
speciality,  half  the  subjects  received  the  interpretative  elicitation 
methodology  and  the  other  half  received  the  generative  elicitation 
methodology.  Supplemental  techniques  were  applied  whenever  time  permitted 
at  Ft.  Carson  and  Ft.  Huachuca,  and  typically  as  a  supplement  to  the 
interpretative  method  (the  generative  method  proved  so  rich  that  it 
consumed  subject's  available  time). 

Knowledge  elicitation  was  conducted  by  two  member  teams.  One  team 
member  served  as  the  principal  knowledge  elicitor  who  asked  the  majority 
of  the  questions.  The  second  team  member  was  a  subject  matter  expert  who 
served  as  a  consultant.  The  consultant's  role  was  to  administer  the 
problems  during  the  problem  solving  technique,  answer  technical  questions 
pertaining  to  the  subject  area,  and  clarify  misunderstandings  that  arose 
between  the  elicitor  and  the  expert. 

The  elicitation  sessions  were  tape  recorded  and  the  tapes  were  later 
transcribed  verbatim.  In  addition,  the  elicitor  took  notes  which  were 
used  to  supplement  the  tapes .  The  transcripts  and  the  notes  were  used  as 
the  source  of  data  in  the  analyses  reported  in  Sections  7  and  8. 
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7.0  DATA  DOCUMENTATION  AND  SAMPLE  DATABASE 


7 . 1  A  Log  of  Data-Collection  Interviews  and  Development  of  Knowledge 
Model 

In  this  section  we  illustrate  our  elicitation  methodologies  by 
presenting  excerpts  of  elicitation  sessions  we  conducted.  These  excerpts 
will  illustrate  the  different  procedures  we  used,  the  questions  and 
follow-ups  that  we  asked  and  how  knowledge  models  were  generated  from  the 
subjects'  responses.  This  section  is  broken  down  by  elicitation 
methodology  (interpretative/generative).  The  format  of  material  presented 
is  as  follows.  Since  the  application  and  analysis  of  each  methodology  is 
the  same  regardless  of  whether  the  situation  assessment  or  order  of  battle 
domain  is  used,  only  traces  for  situation  development  are  presented.  The 
raw  text  is  given.  We  present  the  text  in  essentially  unedited  fashion  as 
it  occurred  in  the  session.  We  do  omit  tangential  remarks  (such  as  if  the 
expert  asks  whether  he  can  mark  on  the  map) .  The  reason  behind  this  is 
that  we  do  not  wish  to  risk  altering  the  expert's  meaning  by  rewording 
what  he  says.  Hence,  the  expert's  responses  may  often  resemble  "stream  of 
consciousness"  thought.  Occasionally,  for  the  sake  of  readability,  we 
include  parenthetical  remarks  to  clarify  what  the  expert  has  said. 

However,  these  remarks  are  always  enclosed  in  brackets  so  that  the  reader 
will  know  which  they  are.  Beneath  the  text  in  italics  are  explanations  as 
to  the  purpose  of  the  questions  asked.  Following  the  text  are  the 
representations  constructed  from  the  text  with  explanations  as  to  how  the 
representations  were  constructed.  Note  that  each  paragraph  in  the  text  is 
numbered  for  purposes  of  reference. 

7.1.1  Interpretative  method. 

Situation  Development  Expert.  The  following  material  is  excerpted 
from  an  interview  with  a  division  G-2.  The  first  part  demonstrates  a 
focused  interview  designed  to  build  a  script  of  the  process  of  situation 
development.  Essentially  the  elicitor  will  be  going  through  his  template 
of  the  script  and  try  to  fill  in  whatever  slots  are  there.  Along  the  way, 
the  elicitor  will  ask  for  elaboration  or  clarification  to  fill  in  details. 
Further,  the  elicitor  will  often  ask  questions  of  the  form,  "Is  there 
anything  else?"  in  order  to  get  as  complete  a  response  as  possible. 

1.  Elicitor:  Now,  let  me  ask  you  initially,  are  there  different  kinds 
of  situation  development  or  conditions  under  which,  say,  the 
procedures  might  be  different,  say,  of  mid- intensity  or 
low- intensity  or  high- intensity  conflict,  or  do  you  basically  go 
through  the  same  types  of  procedures  in  all  different  situations? 

In  other  words,  [with  respect  to]  geographical  locations  or 
intensity  of  conflict,  are  there  different  kinds  of  procedures  that 
you  would  go  through  in  different  kinds  of  situations  or  would  you 
say  that  they're  basically  the  same? 

Question  is  designed  to  get  at  tracks  within  the 

situation-development  script,  i.e. ,  what  conditions  might  affect  how 
situation  development  is  done. 
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2.  Expert:  I  think  there  is  a  basic  process;  I  think  there  are  basic 
variations  of  the  basic  process  with  respect  to  conflict,  and  I  also 
think  that  there  is  probably  a  variation  of  the  process  by  echelon 
where  the  function  is  taking  place. 


3.  Elicitor:  OK,  so  we  have  basically  two  dimensions:  the  conflict 
and  the  echelon  that  you  have  just  mentioned.  Let's  take  them  in 
turn  now.  In  the  conflict,  could  you  say  what  the  different  kinds 
of  conditions  of  conflict  are  that  might  determine  how  you  might  go 
about  developing  the  situation? 

Elicitor  takes  the  expert's  answers  and  asks  him  to  elaborate . 


4.  Expert:  One  of  the  things  we  look  at  frequently  here  is 

low- intensity  conflict  to  develop  the  situation  where  your  principal 
adversary  is  perhaps  a  third-world  military  force  lacking 
sophisticated  weapons,  support  procedures.  Situation  development 
there  becomes  perhaps  less  technical,  more  time-  and 
analytical -intensive,  and  you  pose  that  against  a  high- intensity 
conflict,  you  would  expect  to  bring  to  bear  more  technical  means  and 
a  greater  effort  to  correlate  those  technical  means.  The  analytical 
effort  is  probably  as  great  and  I  think  the  analytical  effort  is 
probably  constant  throughout  the  spectrum  of  conflict,  all  spectrums 
of  conflict.  By  echelon,  the  situation  development  of  the  battalion 
level  is  different  in  terms  of  methodology  than  it  is  at  the 
division  level.  The  battalion- -once  it  is  joined,  post  battle- -its 
effort  is  to  maintain  contact  with  the  enemy  force.  The  [ S ] 2  is  an 
active  player  In  that  the  [S]2  can  certainly  determine  a  great  many 
things  from  force  in  contact- -probably  nothing  more  than  force  in 
contact,  disposition,  morale,  in  some  cases  intent,  and  the  means 
and  method  of  doing  that  probably  don't  vary  a  great  deal.  The 
battalion  [ S J 2  might  want  to  take  prisoners- -I  would  want  to  take 
prisoners.  Hopefully,  we  know  enough  about  the  enemy  [to]  not  just 
snatch  anybody  on  the  battlefield  but  to  go  after  certain  types  of 
prisoners- -squad  leaders,  company  commanders.  With  any  degree  of 
sophistication  and  accuracy  of  intelligence  to  go  after  higher-level 
personnel,  command  posts,  etc.  at  battalion,  regiment  at  a  higher 
level. 


5.  Situation  development  might  certainly  involve  taking  advantage  of 
known  vulnerability,  if  the  enemy  has  a  known  weakness  In  command 
and  control  or  in  command,  control,  and  communications.  Destruction 
of  his  command  post,  destruction  of  his  communications  would  allow 
us  to  do  certain  things  to  exploit  that  vulnerability;  cause  that 
situation  to  be  developed  in  a  method  and  manner  of  our  own 
choosing,  it  is  so  much  simpler  than  getting  inside  that  commander's 

decision  cycle,  not  certainly,  not  his  .  That  is  about  as  much 

as  I  can  elaborate  on  differences  in  spectrum  of  conflict  and 
echelon. 
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6. 


Elicitor:  Let  me  go  on  then,  in  developing  the  situation,  what  are 
your  goals  or  objectives?  What  are  you  trying  to  accomplish  or 
learn  or  do?  What  is  the  purpose  of  doing  the  situation 
development? 

Elicitor  asks  about  goals  relevant  to  situation  development . 


7.  Expert:  The  fundamental  purpose  is  to  determine  the  enemy  intent  to 
get  inside  of  the  enemy  commander's  decision  making  so  that  I  can 
pre-empt  his  plans  and  his  intended  operations --to  cause  an 
initiative  to  move  to  the  United  States  Commander. 


8.  Elicitor:  OK,  here  is  where  I  ask  you  one  of  those  clarifying 
questions .  Why  are  you  trying  to  know  his  intentions  and  plans  and 
so  forth? 

The  question  is  designed  to  get  at  underlying  rationale  for  goals. 
The  elicitor  is  probing  for  an  underlying  mental  model. 

9.  Expert:  To  kill  him  before  he  can  take  any  action  that  would  impact 
on  our  forces. 


10.  Elicitor:  How  does  knowing  his  intentions  allow  you  to  achieve  that 
goal  of  killing  the  enemy  before  he  can  take  action? 

11.  Expert:  That  is  simply  acting  before  he  acts,  striking  him  before 
he  can  strike,  disrupting  his  plan,  train  of  thought,  certainly  his 
intentions,  to  know  what  he  wants  to  do,  when  he  wants  to  do  it,  and 
we're  able  to  pre-empt  that  through  destruction  of  his  force  or  some 
other  means  and  method.  That's  what  we  want  to  do  to  achieve  a 
successful  result. 

12.  Elicitor:  OK,  I  am  going  to  try  to  summarize  what  you  have  just 
said  and  correct  me  if  I  am  wrong.  So,  presumably  the  enemy  has 
some  intend-ed  sequence  that  it  is  going  to  try  to  do;  you  are 
basically  trying  to  learn  that  plan  or  sequence  of  action  the  enemy 
wants  to  take  so  that  you  can  then  intervene.  Does  that  basically 
capture  the  process? 

13.  Expert:  Intervene  before  he  can  put  that  plan  or  that  operation  in 
motion,  before  he  can  initiate  operations. 

14.  Elicitor:  Now,  is  this  a  kind  of  general  principle  of  understanding 
the  enemy's  intention  as  opposed  to  say,  only  in  certain  conditions 
is  this  useful- -that  sort  of  universal? 
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Elicitor  wants  to  determine  whether  the  expert's  answer  depicts  a 
universal  principle  or  is  it  tied  to  some  specific  situation.  This 
will  play  an  important  role  as  to  how  the  knowledge  is  encoded  and 
at  what  level  of  abstraction. 


15.  Expert:  No,  that  is  universal. 


We  now  go  through  the  text  and  illustrate  how  to  build  knowledge 
models  from  the  raw  material.  In  general,  we  took  the  knowledge  structure 
templates  for  scripts,  frames  and  mental  models  and  tried  to  fill  the 
specific  slots  of  those  structures  with  information  obtained  from  the  raw 
materials.  Occasionally  (particularly  the  case  with  frames)  these  will  be 
information  contained  in  the  text  (such  as  when  an  object  is  being 
described)  for  which  there  is  no  slot.  For  example,  in  describing  a 
particular  combat  unit,  reference  might  be  made  to  that  unit's  typical 
mission.  Our  original  version  of  the  frame  template  had  no  slot  for 
mission.  Hence,  in  coding  that  particular  piece  of  data,  we  would  add  a 
slot,  "has  mission,"  to  the  frame  and  fill  in  that  slot  with  the 
information. 

The  decision  of  which  knowledge  structure  to  use  (i.e.,  script, 
frame,  mental  model)  was  guided  by  two  considerations:  first,  certain 
questions  are  asked  explicitly  directed  toward  filling  slots  of  a 
particular  type  of  structure  (see  annotations  to  the  above  interview 
material);  second,  material  is  mapped  to  structures  according  to  what  type 
of  information  it  conveys.  Events  and  goal-oriented  procedures  are  mapped 
to  scripts,  object-related  information  is  mapped  to  frames,  and 
descriptions  of  causal  mechanisms  or  explanations  for  events  and 
procedures  are  mapped  to  mental  models. 

We  start  with  paragraph  1.  We  see  from  the  annotations  to  the 
elicitor' s  question  that  the  elicitor  was  trying  to  get  at  tracks  involved 
in  the  situation- development  script.  Thus  we  start  with  a  script 
template.  The  script  header  is  "situation  development."  We  anticipate 
that  the  subject's  response  will  provide  information  relevant  to  the 
script's  track. 

In  paragraph  2,  the  expert  lists  two  factors- -type  of  conflict  and 
echelon- -that  influence  how  situation  development  is  done.  Hence,  we  list 
these  as  separate  tracks  in  the  situation-development  script  with  the 
paragraph  number  alongside  in  case  we  need  to  recheck  the  text  later  on. 
This  is  illustrated  below: 

Script:  Situation  development 
Tracks:  Type  of  conflict  (2) 

Echelon  (2) 


In  paragraph  3,  the  elicitor  tries  to  flesh  out  the  different  kinds 
of  conflict  and  echelon  that  might  influence  situation  development. 


In  paragraph  4,  the  expert  begins  by  describing  the  difference  between 
low- intensity  and  high- intensity  conflict.  Each  of  these  becomes  an 
elaboration  on  che  type -of -conflict  track  of  the  script,  e.g., 

Type  of  conflict  (2) 

Low- intensity  (4) 

High- intensity  (4) 


In  addition,  the  expert  describes  briefly  how  situation  development 
differs  in  these  two  types  of  conflict;  namely,  that  in  low- intensity 
conflicts,  situation  development  is  less  technical  and  more  time-  and 
analytical-  intensive,  etc.  This  information  can  be  used  to  build  the 
representation  of  the  low- intensity  situation-development  track. 
Unfortunately,  there  is  not  much  detail  given  so  we  would  enter  what  is 
given  and  be  on  the  lookout  for  more  information  later.  Information  that 
is  provided  is  given  below. 

Track:  Low- intensity  conflict  (4) 

Rol  .  Adversary- -third-world  country  (4) 

Props  Weapons  (4) ;  support  procedures  (4) 


This  information  can  also  be  used  to  construct  a  frame  describing 
the  features  of  the  conflict  itself  (as  opposed  to  the  track  which 
describes  the  process  of  situation  development  under  those  conditions) . 

In  order  to  construct  this  frame,  we  would  take  a  frame  template  and  fill 
in  slots. 

First,  we  label  the  frame -ID  "low- intensity  conflict- -third-world 
country"  since  that  describes  what  the  expert  is  talking  about.  We  then 
go  down  through  the  list  of  slots  and  see  which  can  be  filled  from  the 
text.  The  first  slot  is  "frame -of"  which  is  a  description  of  what  the 
frame  is.  In  this  case,  we  are  talking  about  a  battle  with  a  third-world 
country.  The  "is -a"  slot  refers  to  the  more  general  category  of  which  the 
frame  is  a  part.  In  this  case,  battles  are  a  type  of  conflict;  hence, 
"conflict"  would  go  in  the  "is -a"  slot.  The  expert  makes  no  reference  to 
what  the  frame  is  similar  to;  hence,  that  slot  is  left  blank.  For 
"has-parts,"  the  expert  lists  weapons  and  support  procedures  that  make  up 
part  of  the  conflict.  The  "part-of"  slot  refers  to  a  larger  unit  that  the 
current  frame  is  part  of  (e.g.,  a  battalion  is  part  of  a  regiment).  The 
expert  does  not  give  any  information  here,  or  for  the  next  slot  --"size"-- 
either.  The  next  slot- -"descriptions"- -allows  for  a  description  of  the 
frame.  This  might  be  a  list  of  features  which  describe  or  characterize 
the  frame.  One  example  that  the  expert  lists  is  that  the  opponent  is  a 
third-world  military  force.  Hence,  this  feature  would  be  listed  under 
"description."  The  next  slot- -"spatial  configuration”--  remains  unfilled 
since  the  expert  provides  no  information. 

After  this,  the  template  contains  a  place  to  list  features  or 
attributes  associated  with  some  of  the  slot  fillers.  This  is  almost  like 
a  semantic  net- type  representation  format  and,  in  fact,  would  serve  as  a 
good  place  for  frames  and  semantic  nets  to  interface.  In  this  section,  we 
note  that  the  expert  has  characterized  the  third  world's  weapons  and 
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support  procedures  as  being  "not  sophisticated."  Hence,  we  place  the 
following  in  this  section:  weapons  •  not  sophisticated  (4);  support 
procedures  -  not  sophisticated  (4) . 

Finally,  we  note  that  the  expert  compared  the  low-intensity  conflict 
with  high- intensity  conflicts.  Since  our  template  does  not  have  slots  for 
making  comparisons  or  contrasts,  we  generate  a  slot  of  our  own  to  capture 
this  knowledge.  Under  this  "compared  to"  slot,  we  would  have 
"high -intensity  conflict"  (based  on  what  the  expert  said)  and  the 
differences  are:  weapons  are  more  technical  (in  high- intensity  conflict) 
and  support  procedures  are  more  correlated.  A  commonality  is  the 
analytical  effort.  As  a  note,  this  comparison  between  high-  and 
low- intensity  conflicts  can  be  used  to  construct  part  of  a 
"high- intensity"  conflict  frame. 

We  now  present  the  "low- intensity  conflict- -third-world  country 
frame . " 


Frame -ID: 
Frame -of: 
is-a: 

has -parts: 
Description: 
Compared- to : 


Obj  ect/Features : 


Low- intensity  conflict- -third-world  country  (4) 

Battle  (4) 

Conflict  (4) 

Weapons  (4) ,  support  procedures  (4) 

Opponent:  third-world  military  force  (4) 

High- intensity  conflict  (4) 

Differences:  weapons  -  more  technical  (4) 

support  procedures  -  more  correlated  (4) 
Commonalities:  analytical  procedure  (4) 

Weapons  -  not  sophisticated  (4) 

Support  procedures  -  not  sophisticated  (4) 


Having  represented  what  the  expert  said  about  the  different  types  of 
conflict,  we  turn  to  his  discussion  on  the  different  types  of  echelon.  We 
again  return  to  the  tracks  slot  of  the  situation- development  script  which 
is  where  we  anticipate  the  expert  will  be  providing  us  with  information. 

In  fact,  the  expert  makes  reference  to  two  different  echelons  that 
influence  situation  development- -division  and  battalion.  Each  of  these 
then  gets  listed  as  a  subtrack  under  the  echelon  track  of  the 
situation-development  script. 

After  listing  these  two  echelons,  the  expert  begins  describing  the 
situation-development  process  at  the  battalion  level.  This  would  suggest 
creating  a  battalion  situation- development  track  using  a  script  template. 
What  is  different  in  this  situation  is  that  the  expert  is  spontaneously 
generating  this  subscript  (track).  Hence,  the  slots  of  the  template  may 
not  be  filled  in  a  neat  order.  Rather,  the  person  building  the 
representation  may  have  to  go  through  the  text  and  determine  what 
information  goes  in  what  slot. 

First,  however,  the  battalion  track  gets  a  pointer  to  the  battalion 
situation- development  subscript  so  that  the  user  knows  that  there  is  a 
knowledge  structure  being  referred  to.  This  can  be  done  as  follows: 
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Script: 


Situation  development 


Tracks:  Echelon  (2) 

Division  (4) 

Battalion  (4) --pointer  to  Battalion 
S.D.  subscript 

Conflict  (2) 


This  having  been  done,  we  turn  to  the  construction  of  the  battalion 
situation- development  subscript.  We  begin  by  giving  the  subscript  its 
appropriate  label.  The  expert  then  says  that  when  the  battalion  is  joined 
in  battle,  it  maintains  contact  with  the  enemy.  The  rest  of  the 
discussion  centers  on  what  happens  once  this  occurs.  Hence,  the  coming 
into  contact  with  the  enemy  stimulates  the  situation- development  process 
at  the  battalion  level  (according  to  our  interpretation  of  the  text). 
Hence,  the  battalion  coming  into  contact  with  the  enemy  appears  to  be  the 
entry  condition  for  the  subscript,  and  hence,  will  be  listed  under  "Entry 
Conditions"  in  the  template.  The  expert  next  makes  reference  to  the  S-2 
(the  expert  actually  says  "2",  but  the  elicitor  knew  the  expert  meant  S-2; 
otherwise  he  would  have  asked  for  a  clarification) ,  hence  naming  a  role  in 
the  situation-development  process.  The  S-2  is  then  listed  under  roles. 

The  expert  then  listed  some  of  the  things  the  S-2  is  trying  to  do: 
determine  force  in  contact,  disposition,  morale  and  intent.  These  are 
interpreted  to  mean  goals  or  objectives  of  the  situation-development 
process  and  are  therefore  listed  under  the  "goals"  slot. 

The  expert  then  describes  the  actions  involved  in  taking  prisoners. 
He  lists  different  types:  squad  leaders  and  company  commanders.  Hence, 
the  expert  has  identified  new  roles  and  they  will  be  listed  accordingly. 

The  very  act  of  taking  prisoners  represents  procedures  performed  as 
part  of  the  situation- development  process.  This  suggests  a  scene  which  we 
label  "take  prisoners."  The  expert  further  elaborates  on  how  he  would  go 
about  taking  prisoners,  namely  using  "intelligence  to  go  after 
higher- level  personnel,  command  posts,  etc.  at  battalion,  regiment  [and] 
at  a  higher  level."  This  piece  of  knowledge  is  represented  as  an  action 
under  the  "take  prisoners"  scene. 

In  the  next  paragraph  (5),  the  expert  states  that  situation 
development  involves  taking  advantages  of  weaknesses  in  command  and 
control  or  command,  control,  and  communications.  This  gets  listed  as 
another  goal  of  the  subscript. 

Following  this,  the  expert  discusses  how  destroying  enemy  command 
posts  would  favor  the  friendly  side.  Normally,  this  type  of  explanatory 
text  would  suggest  constructing  a  mental  model.  However,  shortly  down  the 
text  the  elicitor  discusses  this  in  more  detail;  hence  the  construction  of 
the  mental  model  is  deferred  until  then.  We,  therefore,  present  the 
battalion  situation- development  subscript  as  developed  up  to  this  point. 
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Subscript:  Battalion  situation  development 
Associated 

Script:  Situation  development 

Goals:  Determine  force  in  contact  (4) 

Determine  enemy's  disposition  (4) 

Determine  enemy's  morale  (4) 

Determine  enemy's  intent  (4) 

Take  advantage  of  weaknesses  in  command  and 
control  or  command,  control  and 
communications  (5). 

Entry 

Conditions:  Battalion  comes  in  contact  with  enemy. 

Rules:  S-2  (4) 

Prisoners  (4)  squad  leaders,  company 
commanders  (4) 

Props:  (No  information  provided) 

Scenes :  Take  prisoners 

Actions:  Use  intelligence  to  go  after  higher-level 

personnel  and  command  posts  at  battalion, 
regiment,  and  higher  echelons 

Continuing  in  the  text  (paragraph  6),  since  the  expert  has  said  that 
he  has  listed  the  differences  in  type  of  conflict  and  echelon,  the 
elicitor  leaves  the  "tracks"  slot  of  the  situation-development  script  and 
goes  on  to  the  "goals"  slot,  asking  the  expert  what  the  goals  are  in 
situation  development. 

In  paragraphs  7  and  9,  the  expert  lists  some  goals  which  are  placed 
in  the  goals  slot: 

Goals:  Determine  enemy's  intent  (7) 

Preempt  enemy  commander's  plans  and  operations  (7) 

Cause  initiative  to  move  to  U.S.  command  (7) 

Kill  enemy  before  he  can  impact  on  friendly  forces  (9) 

In  paragraph  10,  the  elicitor  asks  for  an  explanation  as  to  how 
knowing  enemy  intentions  allows  the  situation  developer  to  achieve  the 
goal  of  killing  the  enemy.  Such  an  explanation- oriented  question  is 
geared  toward  eliciting  a  mental  model.  Hence,  we  will  primarily  be  using 
a  mental  model  template  to  represent  the  expert's  response.  Hence,  we 
will  be  looking  for  references  to  context,  goals,  forces,  objects, 
force/object  interactions  and  outcomes  (note  that  we  mentioned  that  part 
of  this  information  was  given  in  paragraph  7;  hence,  we  will  be  using  some 
of  this  material  to  supplement  the  material  in  paragraphs  11  and  13) . 
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In  looking  for  objects,  explicit  reference  is  made  (paragraph  7)  to 
the  friendly  and  enemy  commanders.  Since  the  situation  developer  is 
discussing  his  role,  he  is  an  implied  player  as  well.  In  paragraph  11,  a 
reference  is  made  to  the  enemy  force;  hence,  that  is  an  object  as  well. 

At  this  point,  a  problem  arises  with  knowledge  representation  in 
general,  and  what  we  seem  to  find- -mental  models --in  particular.  Each 
piece  of  knowledge  is  not  always  stated  explicitly  but  often  is  left 
implicit.  The  "mental  model"  slot  is  a  case  in  point.  The  expert  makes 
reference  to  striking  the  enemy  and  destroying  his  force  (paragraph  11) . 

He  does  not  state  what  will  strike  and  destroy  the  enemy  force.  Normally 
we  would  expect  the  elicitor  to  catch  this  and  ask  a  pertinent  question, 
but  in  real  time,  the  elicitor  will  occasionally  miss  things.  Hence,  the 
person  building  the  representation  may  need  to  fill  in  the  gap.  This  can 
be  done  in  conjunction  with  a  subject-matter  expert  who  would  know  what 
the  missing  information  would  be  (this  would  be  particularly  true  in  cases 
where  the  information  is  not  idiosyncratic).  In  this  case,  the  friendly 
forces  would  be  the  ones  destroying  the  enemy  forces.  We  would  then 
include  "(friendly  forces)"  (11)  in  the  objects  slot.  Parentheses  are 
used  to  enclose  the  information  to  indicate  it  was  inferred  rather  than 
explicitly  stated.  The  number  in  parentheses  alongside  it  indicates  the 
locale  of  the  material  on  which  the  inferred  information  is  based. 

The  goal  of  the  mental  model  is  given  in  paragraph  9- -namely,  kill 
enemy  before  he  can  impact  on  our  forces.  This  goal  is  also  relevant  to 
the  situation-development  script  and  is,  therefore,  encoded  there. 

The  relevant  force  is  also  not  explicitly  stated,  but  Implied  from 
the  goal  of  destroying  enemy  forces.  Here,  the  forces  would  be  friendly 
combat  power.  This  would  be  listed  in  parentheses  under  forces  along  with 
"(11)",  Indicating  it  was  inferred  from  paragraph  (11). 

Next  is  the  interaction  of  objects  and  forces.  In  paragraph  11,  the 
expert  states  that  the  friendly  forces  (implied)  will  strike  the  enemy 
forces  before  they  can  act.  This  gets  listed  under  "force/object 
interactions . " 

Next  is  the  "outcomes"  slot.  In  paragraph  11,  the  expert  lists  a 
number  of  outcomes  that  will  result  from  the  strike.  These  are: 
disruption  of  the  enemy's  plan,  train  of  thought,  intentions,  destruction 
of  the  enemy  force.  In  paragraph  9,  the  expert  implies  that  another 
outcome  would  be  that  the  enemy  would  not  impact  on  friendly  forces.  In 
paragraph  7,  the  expert  states  that  preemptive  enemy  plans  would  cause  the 
initiative  to  move  to  the  friendly  commander.  Hence,  this  Is  another 
outcome  that  gets  listed. 

The  only  slot  that  remains  unfilled  is  the  "context"  slot.  In  order 
to  get  information  relevant  to  this,  the  elicitor  asks  (paragraph  14) 
whether  this  is  a  universal  principle  or  tied  to  specific  conditions.  The 
expert  replies  that  it  is  universal  (presumably  universal  on  the 
battlefield,  although  there  are  undoubtedly  analogies  to  other  contexts). 
Hence,  this  is  filled  in  the  context  slot  of  the  mental  model. 
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The  only  other  thing  to  do  with  regard  to  the  mental  model  is  to 
give  it  a  label  or  name.  This  is  not  really  essential  but  helps  reference 
it  for  purposes  of  discussion.  We  label  this  mental  model  "disrupting 
enemy  forces . " 


We  now  present  the  entire  mental  model  below: 


Mental  Model:  Disrupt  Enemy  Forces 

Context:  Universal  on  battlefield  (15) 


Goal: 


Obj  ects : 


Forces : 


Kill  enemy  before  he  can  take  action  that  would 
impact  on  friendly  forces  (9) 

Enemy  forces  (7) 

(Friendly  forces)  (11) 

Enemy  commander  (7) 

Friendly  commander  (7) 

Friendly  intelligence  officer  (7,  11) 

(Friendly  combat  power)  (11) 


Force/Object 

Interactions:  Friendly  forces  strike  before  enemy  can  act  (11) 


Outcomes:  Enemy  plan  disrupted  (11) 

Enemy  train  of  thought  disrupted  (11) 

Enemy  intention  disrupted  (11) 

Enemy  force  destroyed  (11) 

Enemy  forces  do  not  impact  on  friendly  forces  (11) 
Initiative  goes  to  friendly  commander  (7) 


Once  the  mental  model  is  constructed,  we  link  it  to  the 
representation  that  evoked  it.  Recall  that  this  mental  model  was  invoked 
as  an  explanation  of  why  the  situation  developed  has  the  goal  of  killing 
the  enemy  before  he  can  take  action.  Hence,  a  pointer  to  this  mental 
model  is  attached  to  this  goal  which  is  listed  in  the  goal  slot  of  the 
situation-development  script.  This  is  illustrated  below: 

Script:  Situation  development 

Goals:  Kill  enemy  before  he  can  impact  on  friendly 

forces- -pointer  to  d-M  disrupt  enemy  forces 


This  concludes  our  section  on  knowledge  elicitation  using  the 
interpretative  interview  technique.  We  have  given  examples  as  to  how 
scripts,  mental  models  and  object  frames  can  be  constructed  from  the  raw 
material.  We  have  so  far  presented  a  mental  model  and  a  frame  in  its 
entirety,  but  have  given  just  bits  and  pieces  of  the  ituation-development 
script  we  constructed.  We  now  present  the  unified  situation-development 
script  that  was  constructed  based  on  the  text  provided  above: 
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Script: 


Situation  development 


Tracks:  Echelon  (2) 

Battalion  (4) --pointer  to  battalion 
situation -development  subscript. 
Division  (7) 

Conflict  (2) 

Low  intensity  (4) --pointer  to  low- 
intensity  conflict  frame. 

High  intensity  (4) 

Goals:  Determine  enemy's  intent  (7) 

Preempt  enemy's  plans  and  operations  (7) 
Cause  initiative  to  move  to  U.S.  command  (7) 
Kill  enemy  before  he  can  impact  on  friendly 
forces  - -pointer  to  M-M  disrupt  enemy  forces 

Roles:  Friendly  commander  (7) 

(situation  developer) 


We  now  turn  to  a  demonstration  of  the  second  major  elicitation 
technique  of  the  interpretative  method:  top-down  problem  solving.  This 
technique  is  designed  to  flesh  out  the  procedures  involved  in  the 
performance  of  the  expert's  job,  or  in  terms  of  our  framework,  to  elicit 
the  procedures  to  fill  in  the  script. 

This  technique  is  far  less  structured  than  the  interview  technique 
presented  earlier.  Experts  are  given  a  problem  and  are  asked  to  work  on 
it  as  they  normally  would  while  doing  their  job.  The  elicitor  may  ask 
clarifying  questions ,  but  he  is  not  structuring  the  way  the  expert  solves 
the  problem.  The  elicitor  is  not  telling  the  expert,  e.g.,  "now  consider 
the  goal  aspects  of  what  you're  doing."  Therefore,  the  elicitor  does  not 
have  this  a  priori  expectancy  as  to  what  kinds  of  knowledge  the  expert 
will  emit.  The  elicitor  must  infer  from  the  content  (and  responses  to 
clarifying  questions)  what  the  expert  is  doing  and  what  knowledge  he  is 
using.  It  is  useful,  therefore,  fox  the  data  analyst  to  have  either 
sufficient  knowledge  of  the  domain  to  make  inferences  where  necessary  or 
to  have  a  domain  expert  help  with  the  analysis. 

As  with  the  interview  text,  we  begin  by  presenting  a  trace  of  the 
problem-solving  part  of  the  session,  followed  by  a  discussion  of  how 
knowledge  representations  were  built  from  the  text. 

1.  Expert:  The  first  thing  that  I  would  want  to  do  as  an  assistant  G-2 
operations  officer  is  get  out  of  the  TOC  and  to  talk  to  the  G-2 
operations  officer  of  the  defending  division.  That  one-on-one  would 
give  me  a  first-hana  assessment  of  how  they  view  what's  going  on  out 
there.  If  time  continues  to  permit  me  to  continue  to  do  personal 
contact  type  assessments,  I  would  want  to  talk  to  the  brigade  S-2s 
and  the  brigade  commanders  of  the  units  in  contact.  And,  in  that 
process,  I  would  want  to  do  some  kind  of  limited  reconnaissance. 
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Expert  has  been  briefed  on  the  background  of  his  problem.  He  has 
been  given  a  mission,  intelligence  reports,  and  a  map  with  enemy 
order  of  battle  posted.  Expert  is  asked  how  he  would  proceed  from 
there . 


2.  Military  Consultant:  [You  want  to  get  information  from]  units  on 
line  that  [others  will]  pass  through? 


3.  Expert:  Yes. 


4.  Elicitor:  What  are  you  going  to  be  asking  them? 


5.  Expert:  Their  assessment  of  the  unit's  actions  along  that  first 
defensive  belt;  what  are  they  doing  and  how  do  they  do  it;  what  is 
their  morale  like?  Indications  if  there  might  be  some  kind  of 
logistics  problems  and  resupply.  I'll  expect  the  first  defensive 
belt  of  the  31st  MRD  probably  be  supplied  better  than  the  46th. 
[Elicitor  asks  Expert  to  role  play.]  I  would  first  want  to  look  at 
the  brigade  that  holds  the  first  avenue  of  approach  through  the  31st 
MRD.  Avenue  of  approach  1  though  the  31st  GMRD. 

6.  Military  Consultant:  I  noted  that  he  is  assessing  avenues  of 
approach.  He  sketched  them  out  on  the  map.  This  would  be  something 
you  would  do  to  get  yourself  up  to  speed  or  get  the  background 
information  on  the  scenario. 

Elicitor  number  2  (the  military  consultant )  summarizes  what  the 
expert  was  doing  while  there  was  no  discussion  taking  place  and  then 
asks  the  expert  why  he  is  doing  it. 

7.  Expert:  I  would  want  to  get  the  G-2s  from  the  23rd  Division  (on 
line)  assessment  of  the  situation.  The  23rd  [Division] ,  in 
anticipation  of  future  operations,  has  done  an  IPB  and  looked  at 
axes  of  advance  into  the  31st  division  sector.  If  I  know  it  at  the 
time,  I  will  try  to  get  specific  information  about  specific  passage 
points  that  my  battalions  and  brigades  would  have  to  come  through. 
Their  immediate  concerns  are  going  to  be  what's  the  threats  and 
what's  the  resistance  they  will  have  to  face  as  they  go  through 
those  passage  points  and  achieve  initial  contact  with  the  enemy 
forces . 


8.  Military  Consultant:  As  the  G-2  of  the  23rd,  I  would  tell  you  that 
we  have  observed,  uh,  haven't  sorted  out  what  all  the  activity's 
that's  going  on  out  here  is,  but  there's  a  lot  of  activity  and  we 
agree  that  it  appears  that  they  are  digging  in  on  the  defense.  We 
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think  this  is  probably  the  security  area  and  we  are  trying  to 
determine  where  they  are  setting  up  ambushes  or  killing  zones.  We 
are  developing  that  chain  of  events. 

The  military  elicitor  role  plays  the  person  that  the  expert  says  he 
would  consult.  A  key  feature  of  the  problem-solving  technique  is 
that  the  expert  will  often  want  to  interact  with  others  to  get  vital 
information.  It  is  useful,  therefore,  to  have  someone  in  these 
other  roles.  This  could  either  be  other  experts  (in  which  case  the 
knowledge  elicitation  may  focus  on  a  group  problem-solving  session) 
or  a  domain  consultant ,  as  in  this  case,  which  can  simulate  the 
other  roles. 


9.  Expert:  I've  got  to  assume  that  we  have  this  mission  and  my 

division  is  going  to  go  through  a  deliberate  set  planning  process. 
That  means  that  the  G-3  and  I,  my  G-2,  and  the  division  G-3  are 
going  to  recommend  courses  of  action  to  the  commander. 

10.  Expert:  The  first  intelligence  assessment  that  we  have  to  take  into 
account  is  relative  strength.  We  have  a  general  idea  of  the 
relative  strength  that  is  approximately  70%.  The  key  thing  we  want 
to  do  is  achieve  surprise,  achieve  breakthrough  at  that  first 
defensive  belt.  We  want  to  do  it  in  such  a  manner  that  will  cause 
enough  disarray  in  the  first  defensive  belt  to  catch  the  supporting 
units  that  are  behind  that  first  defensive  belt  in  a  situation  where 
they  could  not  respond  immediately,  be  that  response  movement 
forward  or  in  this  case,  looking  at  what's  in  the  rear  here, 
probably  retrograde  movement  to  implement  the  defensive  sector  of 
the  46th  motorized  rifle  division. 


11.  Elicitor:  So  you're  setting  up,  it  sounds  like,  the  goals  that 
you're  going  to  try  to  achieve,  or  in  other  words,  ... 

Elicitor  tries  to  abstract  the  purpose  behind  what  the  expert  is 
doing . 


12.  Expert:  Recommending  courses  of  action. 


13.  Elicitor:  Right,  but  goals  that  you  want  these  courses  of  action  to 

satisfy. 

14.  Expert:  Yes,  now  between  the  [G]-3  and  I,  what  I  would  recommend  to 
the  [ G ] - 3  is  that  we  attempt  to  achieve  the  first  breach  through  a 
combination  of  joint  air  attack  and  ground  attack  and  then  we  try  to 
achieve  it  on  what  this  is  the,  call  that  avenue  of  approach  1. 

It's  clear  it's  the  best  one  that's  there,  which  represents  the 
secondary  avenue  of  approach,  probably  given  what  our  mission  is 
it's  to  achieve  objective  1.  I'm  less  concerned,  I  would  suggest  to 
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Che  [G]-3  Chat  he  asks  for  perhaps  a  change  in  boundary  between  Che 
54th  and  53rd,  unless  Che  53rd  is  also  going  Co  conduct  a  supporting 
attack  or  they  have  the  main  attack.  I  think  it's  important  that  we 
have  control  of  the  high  speed  avenues  of  the  approach.  Those  are 
just  considerations  for  the  [G]-3  what  he  needs  to  look  at  and 
there's  an  interaction  between  the  [G]-2  and  the  [G]-3  at  this  point 
in  time,  kind  of  loses  its  staff  parochialness  or  part  of  a  specific 
function  in  as  much  as  the  [ G ] - 2  is  going  to  talk  about  things  that 
address  the  [G]-3's  functional  area,  the  [G]-3  is  going  to  talk 
about  things  that  are  going  to  address  the  [G]-2's  functional  area 
and  there  is  interaction  there.  1  would  say  that  those  traits 
you'll  find  ought  to  be  characterizing  good  staff  relationships 
between  [G]-2s  and  [G]-3s.  The  current  Chief  of  Staff  of  the  Army 
once  told  me  when  I  worked  for  him  that  [G]-2s  and  [G]-3s  ought  to 
be  interchangeable.  That,  I  think,  is  certainly  a  desirable 
capability. 


In  paragraph  1,  the  expert  has  just  been  given  an  overview  of  the 
problem  (presented  in  Appendix  C) .  In  response  to  being  asked  how  he 
would  work  on  the  problem,  the  expert  gives  different  actions  he  would 
take:  personal  interactions  with  other  units  and  limited  reconnaissance . 
These  general  actions  suggest  different  scenes  for  a  situation- development 
script. 

They  belong  in  scripts,  as  opposed  to  frames  or  mental  models  since 
they  describe  procedures.  They  are  scenes  as  opposed  to  actions,  since 
they  describe  generalized  actions  for  which  further  subactions  would  be 
required  to  specify  how  to  carry  these  out  (e.g.,  "doing  limited 
reconnaissance"  is  not  detailed  enough  to  tell  how  this  should  be  done). 

The  above  describes  rules  for  distinguishing  scenes  from  actions. 

It  is  also  necessary  to  distinguish  scenes  from  each  other  (e.g.,  what 
constitutes  a  separate  scene  vs.  part  of  the  same  scene).  Scenes  are 
typically  characterized  by  their  setting,  the  players  involved,  and  the 
goals  involved.  Hence,  although  talking  to  division  intelligence  officers 
and  brigade  intelligence  officers  may  involve  similar  objectives,  the 
settings  and  players  involved  are  quite  different. 

Hence,  talking  to  the  G-2  operations  of  the  defending  division  and 
talking  to  the  on-line  brigade  personnel  would  be  separate  scenes.  It  is 
difficult  to  determine  from  the  expert's  response  whether  talking  to 
brigade  S-2's  and  brigade  commanders  should  be  represented  as  separate 
scenes,  since  the  context  and  goals  may  be  the  same  for  both.  A  general 
principle  which  Schank  (1982)  argues  for  is  that  pieces  of  knowledge  may 
be  stored  in  one  common  structure  until  enough  new  information  comes  in  to 
differentiate  those  pieces  of  knowledge,  necessitating  separate  structures 
to  represent  them.  We  will  adopt  this  philosophy  here,  assuming  that 
talking  to  brigade  S-2s  and  commanders  is  part  of  a  more  general  scene, 
"talking  to  brigade  personnel"  wi'_.i  "talking  to  brigade  S-2s"  and  "talking 
to  brigade  commanders"  as  separate  actions  in  that  scene.  If  the  expert 
elaborates  on  these  two  actions  in  such  a  way  as  to  suggest  that  they  are 
separate  scenes  (e.g.,  goals  and  contexts  are  different),  then  they  can  be 
re-represented  as  separate  scenes. 
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To  summarize  then,  we  would  take  the  content  of  paragraph  1  and 
create  three  scenes:  talk  to  G-2  operations  of  the  defending  division, 
talk  to  on-line  brigade  personnel,  and  conduct  limited  reconnaissance. 
Under  "talk  to  G-2  operations,"  we  have  a  specific  action  which  is  "get 
their  assessment  of  what's  going  on."  Under  "talk  to  on-line  brigades," 
we  have  actions  of  "talk  to  commander,"  "talk  to  S-2s,"  and  subactions  of 
"get  their  assessments"  under  each.  We  illustrate  this  below: 

Script:  Situation  development 

Scene:  Talk  to  defending  division's  G-2 

operations  (1) 

Actions:  Get  assessments  (1) 

Scene:  Talk  to  on-line  brigade  personnel  (1) 

Actions:  Talk  to  S-2  (1) 

Get  assessments  (1) 

Talk  to  commander  (1) 

Get  assessments  (1) 

Scene:  Conduct  limited  reconnaissance  (1) 


In  the  process  of  generating  scenes,  the  expert  has  named  other 
roles  that  occur  in  the  script,  namely,  defending  division  G-2  operations, 
brigade  S-2,  and  the  brigade  commander.  If  these  are  not  already  listed 
under  the  scripts  role  slots,  we  would  add  them  there. 

In  paragraph  4,  the  elicitor  wants  to  flesh  out  some  of  the  actions 
that  might  be  involved  when  the  expert  Is  talking  to  other  personnel. 
Hence,  his  question  is  directed  at  just  that. 

In  paragraph  5,  the  expert  elaborates  on  what  he  would  talk  to  the 
others  about.  He  states  he  would  want  their  assessments  of  what  the  enemy 
units  in  the  first  defensive  belt  were  doing  and  how  they  were  doing  it. 

He  would  also  want  to  know  what  their  morale  is  like  and  whether  there  are 

indications  of  logistics  and  supply  problems. 

The  expert  makes  the  comment  that  he  would  expect  the  first 
defensive  belt  of  the  31st  MRD  (Motorized  Rifle  Division)  to  be  better 
supplied  than  the  46th  MRD.  Although  the  reason  for  this  is  not  stated  in 

the  text,  the  expert  has  received  the  PERINTREP  (see  the  problem  presented 

in  Appendix  C)  of  the  23rd  Armor  Division  (the  division  that  is  defending) 
which  contains  the  information  that  the  31st  MRD  has  priority  of  supply. 
Hence,  the  expert  is  undoubtedly  making  the  inference  that  the  31st  MRD 
will  be  better  supplied  than  the  46th  from  this  piece  of  information. 

This  inference  is  difficult  to  deal  with.  There  are  a  number  of 
possible  representations  for  this.  Perhaps  the  simplest  is  using  a 
production  rule  format  "If  unit  gets  priority  of  supplies,  then  it  will  be 
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better  supplied  than  unit  that  does  not  get  priority."  Production  rules 
are  not  formally  part  of  an  "interpretative"  representation  scheme, 
although  they  are  certainly  readily  incorporable  into  them. 

Another  way  to  represent  this  would  be  through  a  "supply"  script, 
which  describes  how  units  get  supplied.  The  outcome  of  such  a  script 
would  be  that  units  have  supplies.  Hence,  units  run  through  this  script 
(e.g.,  those  that  get  priority  of  supplies)  would  be  better  supplied  than 
units  that  don't  get  run  through  the  script  (e.g.,  those  that  do  not  get 
priority  of  supplies).  From  the  information  given  in  the  text,  it  is 
impossible  to  tell  how  the  expert  made  this  inference. 

If  it  were  essential  for  the  elicitor  to  focus  on  this  particular 
inference,  he  would  have  asked  a  follow-up  question  to  get  the  answer. 
However,  the  elicitor  was  focusing  on  the  actions  the  expert  would  take 
while  talking  to  personnel  of  other  units.  Hence,  pursuing  this  inference 
may  have  distracted  the  expert.  Since  the  expert  himself  did  not  pursue 
this  inference,  the  elicitor  decided  not  to  disrupt  the  expert's  train  of 
thought  (generally  regarded  as  good  advice  while  conducting  knowledge 
elicitation).  Finally,  the  expert  mentions  that  he  would  want  to  look  at 
the  brigade  that  holds  the  first  avenue  of  approach  through  the  31st  MRD. 

In  paragraph  6 ,  the  military  consultant  observes  that  the  expert  is 
actually  studying  the  map.  This  offers  a  new  source  of  data,  what  the 
subject  actually  does,  in  addition  to  what  he  says.  Here,  the  expert  is 
carrying  out  the  actions  he  last  talked  about  in  paragraph  5,  i.e., 
looking  at  avenues  of  approach. 

To  represent  the  text  in  paragraphs  5  and  6,  we  note  that  the  expert 
has  added  new  actions  relating  to  the  scenes  involving  talking  to 
personnel  in  other  units  and  has  also  generated  a  new  scene- -looking  at 
avenues  of  approach.  We  add  this  information  to  the  script: 

Script:  Situation  development 

Scene:  Talk  to  defending  division's  G-2  operations  (1) 

Actions:  Get  assessment  of  what  enemy  units 

in  first  (5)  defensive  belt  are  doing 
and  how  they  are  doing  it  (5). 

Get  assessment  of  enemy  morale  (5). 

See  if  indications  of  logistics  and 
resupply  problems  (5). 

Scene:  Talk  to  on-line  brigade  personnel  (1) 

Actions:  Talk  to  S-2  (1) 

Talk  to  commander  (1) 

Get  assessments  of  what  enemy  units 
in  first  (5)  defensive  belt  are  doing 
and  how  they  are  doing  it  (5) . 
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GeC  assessment  of  enemy  morale  (5). 
See  if  indications  of  logistics  and 
resupply  problems  (5). 


Scene:  Conduct  limited  reconnaissance  (1) 

Scene:  Look  at  avenue  of  approaches  through  area  of 

mission  (5) 

Action:  Select  specific  area  of  approach  (5) 


Note  that  in  the  last  scene  (look  at  avenues  of  approach) ,  we 
generalized  it  from  the  specific  cases  (i.e.,  the  31st  MRD  sector,  first 
avenue  of  approach)  to  the  generic  case  area  where  mission  is  select 
avenue  of  approach.  Through  familiarity  with  the  scenario,  the  person 
building  the  knowledge  representation  knows  that  the  expert's  mission  is 
in  the  section  of  the  31st  MRD  sector.  We  would  not  expect  the  expert  to 
always  look  at  that  sector  (e.g.,  if  the  expert  were  fighting  in  Korea 
instead  of  Germany,  he  would  not  care  about  the  31st  MRD  sector,  which  is 
in  Germany) .  Having  a  domain  expert  as  a  consultant  while  building  such 
representations  will  help  check  the  validity  of  such  generalizations. 
Ideally,  however,  the  elicitor  would  ask  the  expert  being  interviewed  to 
supply  such  generalizations.  Unfortunately,  it  is  impossible  from  a 
practical  point  of  view  to  have  the  expert  clarify  everything  he  says. 

In  paragraph  7,  the  expert  discusses  the  desire  to  talk  to  the  G-2 
of  the  division  currently  on-line  and  also  what  he  would  like  to  talk  to 
him  about.  What  is  interesting  here  is  that  the  expert  has  made  another 
reference  to  talking  to  personnel  of  the  on-line  division  (the  previous 
reference  was  talking  to  the  G-2  operations).  This  suggests  we  need  to 
generalize  these  two  actions  into  a  common  scene,  namely  "talk  to  on-line 
division  personnel"  (similar  to  "talk  to  on-line  brigade  personnel")  which 
replaces  "talk  to  defending  division's  G-2  operations."  This  latter  scene 
now  becomes  an  action  sequence  in  the  more  general  scene.  The  new  scene 
is  presented  below. 

Scene:  Talk  to  on-line  division  personnel 

Actions:  Talk  to  G-2  operations 

Get  assessment  of  what  enemy  units  in 
first  defensive  belt  are  doing  and  how 
they  are  doing  it  (5). 

Get  assessment  of  enemy  morale  (5). 

See  if  indications  of  logistics  and 
resupply  problems  (5). 

Talk  to  G-2  (7). 

See  what  threats  and  resistance  own 
battalions  and  brigades  will  face  as 
they  pass  through  axes  of  advance  into 
enemy  territory  (7). 
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Note  that  the  expert  has  named  the  on-line  division's  G-2  as  a  new 
role.  This  would  be  added  to  the  roles  slot,  if  not  already  there. 

In  paragraph  8,  the  elicitor's  military  consultant  role  plays  the 
G-2  that  the  expert  wants  to  talk  to.  Since  military  problem  solving  is  a 
group  process,  it  adds  realism  and  credibility  to  our  elicitation  if  we 
can  provide  the  additional  roles  while  the  expert  works  through  the 
problem.  Therefore,  it  is  useful  to  have  an  expert  consultant  when 
utilizing  this  technique. 

In  paragraph  9,  the  expert  appears  to  make  reference  to  a  new 
script:  the  division  planning  process.  This  would  suggest  a  tentative 

construction  of  a  new  script  although  much  more  information  is  needed.  An 
entry  condition  appears  to  be  specified- -having  a  mission.  The  roles  of 
G-3,  G-2,  G-2  operations  (the  expert  referring  to  "I"),  and  the  commander 
are  named.  Finally,  the  reference  to  recommending  courses  of  action  to 
the  commander  suggests  either  a  scene  or  an  outcome  depending  upon  whether 
it  Is  a  process,  a  result  of  a  process,  or  both.  More  information  is 
needed  to  clarify  this . 

Situation  development  is  part  of  the  overall  division-planning 
process.  Hence,  this  expert  may  be  explicitly  looking  at  his  situation 
development  "script"  in  the  context  of  a  much  larger  one.  This  is  very 
noteworthy  and  in  looking  at  the  rest  of  the  text  we  will  be  interested  in 
seeing  if  this  hypothesis  proves  to  be  correct. 

In  paragraphs  10-13,  the  expert  states  explicitly  that  he  is 
thinking  in  terms  of  this  larger  script- -namely ,  recommending  courses  of 
action  and  listing  the  objectives  these  courses  should  satisfy.  Hence,  we 
begin  to  construct  this  new  script,  "division  planning  process,"  from  the 
material  in  paragraphs  9-13. 

To  reiterate  from  paragraph  9,  we  have  the  script  header,  the  entry 
condition  (having  a  mission),  some  roles  (G-3,  G-2,  G-2  operations, 
commander) ,  and  the  goal/action  of  recommending  courses  of  action  to  the 
commander.  In  paragraph  10,  the  expert  states  that  the  first  intelligence 
assessment  he  would  want  to  take  into  account  is  relative  strength  of  the 
enemy.  This  suggests  two  things.  First,  part  of  the  planning  procedure 
includes  taking  into  account  the  enemy's  strength.  Second,  since  the 
expert  discusses  the  enemy's  strength,  it  dan  be  inferred  that  thi^  jets 
computed  somewhere.  The  question  remains  whether  this  is  done  as  part  of 
the  planning  script  or  the  situation-development  script.  The  expert 
offers  a  clue  here,  by  referring  to  the  enemy's  strength  as  an 
"intelligence  assessment"  (paragraph  10).  This  suggests  that  the 
production  of  the  estimate  may  in  some  sense  be  part  of  the  situation- 
development  process,  while  the  use  of  the  estimate  may  involve  other  parts 
of  the  planning  process. 

Hence,  we  would  want  to  update  our  situation-development  script  to 
include  "compute  relative  strength  of  enemy"  as  either  a  scene,  an  action 
within  a  scene,  or  (as  turns  out  to  be  correct)  an  additional  script 
(associated  with  the  analysis  of  enemy  order  of  battle)  which  is  one  step 
below  situation  development  in  a  script  hierarchy.  More  information  would 
be  required  to  determine  exactly  where  this  would  fit.  Similarly,  some 
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actions  in  the  planning  script  would  include  the  information  regarding 
enemy  strength.  This  would  be  stored  with  a  pointer  to  the  place  in  the 
situation- development  script  where  this  number  is  computed. 

Unfortunately,  the  expert  does  not  explicitly  elaborate  on  how  the 
enemy  strength  number  is  to  be  used.  However,  we  can  make  some  inferences 
(which  domain  experts  could  evaluate)  based  on  the  text  that  follows  in 
paragraphs  10-14.  In  the  rest  of  paragraph  10,  the  expert  lists  some 
goals  he  would  like  to  achieve  (surprise,  breakthrough  of  first  defensive 
belt  causing  disarray  so  that  supporting  units  cannot  respond) .  In 
paragraph  14,  the  expert  outlines  parts  of  a  plan  for  achieving  these 
objectives.  Presumably,  the  strength  of  the  enemy  plays  a  role  in  how 
well  one  can  achieve  these  goals . 

Getting  back  to  paragraph  10,  in  addition  to  listing  his  objectives, 
the  expert  presents  a  mini- scenario  of  a  surprise  breakthrough  which 
causes  disarray  and  prevents  supporting  units  from  responding.  This 
suggests  a  mental  model.  Using  our  template,  we  get  an  inferred  context 
of  the  battlefield.  The  objects  are  friendly  forces  (implied  by  expert's 
reference  to  "we"),  enemy  first  defensive  belt,  and  enemy  supporting 
units.  Forces  include  strength  of  enemy  and  strength  of  friendly  forces 
(implied) .  The  goals  are  to  achieve  surprise  and  achieve  successful 
breakthrough  (which  includes  a  subgoal  of  having  supporting  units  not 
respond).  The  force  and  object  interactions  include  friendly  force 
catching  enemy  force  unprepared  (implied  by  "surprise")  and  friendly  force 
overpowering  enemy  force  (implied  by  "achieve  breakthrough").  Enemy 
strength  might  be  used  in  computing  this  event.  This  could  be  computed 
using  a  formula  or  another  mental  model.  We  would  need  more  information 
to  determine  this.  The  outcomes  are  breakthrough  achieved,  and  enemy 
supporting  units  incapable  of  responding.  We  present  the  representation 
of  this  mental  model  below. 

Mental  Model:  Surprise  Breakthrough 

Context:  (Battlefield)  (10) 

Goals:  Achieve  surprise  (10) 

Achieve  successful  breakthrough  (10) 

Subgoal:  Prevent  enemy  supporting  units  from 

assisting  (10) 

Objects:  (Friendly  forces)  (10) 

Enemy  first  defensive  belt  (10) 

Enemy  supporting  units  (10) 

Forces:  Enemy  forces  strength  (10) 

(Friendly  forces  strength)  (10) 

Force/Object 

Interactions:  (Friendly  forces  catch  enemy  forces 

unprepared)  (10) 

(Friendly  forces  overpower  enemy  forces)  (10) 
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Outcomes:  Friendly  forces  achieve  breakthrough  (10) 

Enemy  supporting  units  Incapable  of 
responding  (10) 

In  addition  to  this  mental  model,  we  have  some  information  relevant 
to  the  division-planning  script.  We  outline  the  beginnings  of  this 
script: 

Script:  Division  planning 

Goals:  Recommend  courses  of  action  to  the 

commander  (9) 

Subgoal:  Achieve  surprise  (10) 

Achieve  breakthrough  (10) 

Entry 

Conditions:  Have  mission  (9) 

Roles:  G-2  (9),  G-3  (9),  G-2  operations  (9), 

commander  (9),  enemy  first  defensive 
belt  (10),  enemy  supporting  forces  (10) 

Scenes/ 

Actions:  Assess  enemy  strength  (10) - -pointer  to 

situation -development  script,  scene/action, 
compute  enemy  strength. 

Set  objectives  (10) 

Paragraph  14,  in  essence,  contains  two  sections.  In  the  first 
section,  the  expert  begins  to  outline  a  plan.  In  the  second  section,  he 
discusses  the  G-2  and  G-3  roles.  In  the  first  part  of  the  ;aragraph,  the 
expert  recommends  a  joint  air  and  ground  attack.  He  also  selects  a 
particular  avenue  of  approach  to  conduct  the  attack  on.  Each  of  these 
suggests  a  scene,  e.g.,  "determine  composition  of  attacking  units"  and 
"select  avenue  of  approach."  The  expert  does  not  elaborate  on  the 
selection  of  the  units,  but  does  discuss  selecting  the  avenue  of  approach. 
He  selects  a  particular  one,  avenue  of  approach  1  (which  is  in  the 
southern  portion  of  the  enemy  unit  he  will  be  attacking),  which  he  labels 
the  best.  Since  it  is  a  high-speed  avenue  of  approach,  he  recommends  the 
attack  pass  through  there.  In  order  to  get  this  avenue,  he  must  request 
that  the  G-3  ask  for  a  change  in  boundary  since  the  avenue  is  part  of 
another  division  sector.  He  will  do  this  unless  the  other  division  needs 
to  use  that  avenue . 

In  representing  all  this,  we  have  under  the  scene  "select  avenue  of 
approach,"  the  action,  "look  for  high-speed  avenue  of  approach."  We  would 
still  need  information  to  determine  what  makes  an  avenue  high  speed.  It 
turns  out  that  the  avenue  the  expert  refers  to  is  a  major  highway.  This 
could  be  represented  in  semantic  net  format  (although  such  a 
representation  is  not  technically  "interpretative");  highway  "isa" 
high-speed  avenue  of  approach. 
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The  requesting  of  the  boundary  change  really  is  a  rule  as  to  what 
should  be  done  if  the  avenue  of  approach  is  not  part  of  the  division's 
sector.  This  can  be  represented  in  production- rule  format  (again, 
technically  not  an  "interpretative"  method):  "If  desired  avenue  of 
approach  is  not  in  one's  sector  and  is  not  needed  by  unit  whose  sector  it 
is  in,  then  ask  G-3  to  request  a  boundary  change." 

The  expert  next  discusses  the  interaction  between  the  G-2  and  the 
G-3  and  that  they  will  address  each  other's  functional  area.  This  could 
be  represented  in  one  of  two  ways.  First,  it  can  suggest  that  the  actions 
performed  by  the  G-2  and  G-3  can  have  either  one  as  the  actor  as  opposed 
to  assigning  specific  tasks  to  each.  Second,  it  could  suggest  a  separate 
interaction  scene  where  the  G-2  and  G-3  interact  and  exchange  information 
relevant  to  the  other's  area.  This  seems  to  be  closer  to  what  the  expert 
is  saying.  Hence,  we  create  a  new  scone  with  G-2  and  G-3  interactions 
where  G-2  addresses  G-3's  functional  area  and  G-3  addresses  G-2's 
functional  area.  We  add  new  scenes  to  the  planning  scripts  and  get  the 
following. 

Script:  Division  planning 

Goals:  Recommend  courses  of  action  to  the  commander  (9) 

Subgoals:  Achieve  surprise  (10) 

Achieve  breakthrough  (10) 

Entry 

Conditions:  Have  mission  (9) 

Roles:  G-2  (9),  G-3  (9),  G-2  operations  (9), 

commander  (9) 

Enemy  first  defensive  belt  (10) 

Enemy  supporting  forces  (10) 

Scenes:  Assess  enemy  strength  (10) - -pointer  to  situation 

Development 

Script:  Scene/action  compute  enemy  strength 

Set  objectives  (10) 

Select  avenue  of  approach  (10) 

Actions:  Look  for  high  speed  avenue  of  approach  (14) 

("isa"  highway)  (14) 

If  desired  avenue  of  approach  is  not  in  one’s 
sector  and  is  not  needed  by  unit  whose  sector  it 
is  in,  then  ask  G-3  to  request  boundary 
change  (14) . 

G-3/G-2  interaction  (4) 

Actions:  G-3  addresses  functional  area  of  G-2  (14) 

G-2  address  functional  area  of  G-3  (14) 
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Finally,  the  expert  briefly  describes  the  "interchangeability"  of 
the  G-2  and  the  G-3.  This  is  really  giving  a  description  of  the 
capabilities  (or  desired  capabilities)  of  each  person.  Such  descriptions 
are  most  suitably  captured  by  frames.  However,  this  information  does  not 
seem  appropriate  for  the  slots  we  have  in  the  frames  we  have  been  using 
(e.g.,  "spatial  configuration").  Hence,  we  propose  to  construct  a  "person 
frame"  which  represents  information  about  people.  We  can  include  slots 
like  training,  experience,  tasks  performed,  capabilities,  etc.  Based  on 
the  text,  we  have  some  information  for  these  slots.  We  already  know  that 
the  G-2  and  G-3  recommend  courses  of  action  to  the  commander;  hence,  this 
would  be  a  "task  performed"  slot  member.  Similarly,  addressing  each 
other's  functional  areas  would  be  tasks  performed  by  the  G-2  and  G-3.  The 
interchangeability  of  G-2s  and  G-3s  suggests  capabilities  of  each  to 
perform  the  other's  jobs.  Hence,  we  construct  G-2  and  G-3  frames  as  such: 


Person  Frame:  G-2 

Tasks  Performed:  Recommend  courses  of  action  to 
commander 

Address  G-3  functional  area 
Capabilities:  Can  perform  G-3  role 

Person  Frame:  G-3 


Tasks  Performed:  Recommend  courses  of  action  to 
commander 

Address  G-2  functional  area 


Capabilities : 


Can  perform  G-2  role 


There  is  one  final  thing  to  note  about  paragraph  14.  The  expert 
makes  reference  to  the  Chief  of  Staff  of  the  Army.  Note  that  we  have  not 
included  him  in  any  role  in  either  of  the  scripts  we  develop.  This  is 
because  the  expert  did  not  name  him  as  a  player  in  any  process,  but  rather 
refers  to  a  remark  made  by  him.  Hence,  the  knowledge  that  there  is  a 
Chief  of  Staff  of  the  Army  is  ignored  unless  new  information  is  added  that 
he  plays  a  key  role  in  our  process. 

7.1.2  Generative  method. 

Situation  Development  Expert.  The  following  material  is  excerpted 
from  a  generative  problem-solving  session  with  the  he -d  of  a  division 
all-source  production  section.  The  generative  or  data-driven 
problem-solving  technique  is  designed  as  a  method  for  investigating  how 
experts  arrive  at  a  picture  or  solution  to  a  problem  by  responding  to 
individual  pieces  of  data,  integrating  them,  and  drawing  inferences  and 
forming  hypotheses  as  they  proceed  through  the  problem.  This  technique 
serves  several  purposes:  it  allows  for  the  elicitation  of  detailed 
knowledge  and  situation  specific  procedures;  it  allows  for  the  elicitation 
of  data-driven  inference  rules;  and  it  helps  elicit  the  procedures  experts 
use  to  integrate  data. 
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The  questions  the  elicitor  asks  are  driven  toward  these  goals.  The 
elicitor  will  ask  the  expert  what  pieces  of  data  mean  to  him  in  order  to 
elicit  inference  rules.  This  will  be  used  to  construct  production  rules 
where  the  antecedent  conditions  are  those  the  expert  comes  up  with.  The 
expert  may  ask  for  critical  features  of  the  expert's  rules  or  descriptions 
of  the  concepts  he  is  using.  This  will  be  useful  in  constructing  semantic 
network- type  representations.  The  elicitor  may  also  ask  for  underlying 
rationales  for  the  expert's  procedures  and  rules  so  that  mental  models  can 
be  constructed  (much  the  same  as  in  the  interpretative  method) . 

As  with  the  interpretative  method,  we  begin  by  presenting  excerpts 
from  the  elicitation  session  and  then  show  how  the  material  is  used  to 
build  knowledge  models. 

1.  Elicitor:  We  are  going  to  be  giving  you  information  today  on 
situational  development  that  is  not  exactly  like  what  you  would 
normally  see.  Please  bear  with  us  and  you  do  your  best  to  develop  a 
picture  of  the  situation.  What  I  am  going  to  do  is  give  you  pieces 
of  information,  piece  by  piece,  and  start  by  making  no  assumptions 
about  anything  and  then  as  you  gather  more  and  more  information  just 
please  let  us  know  what  picture  you're  putting  together.  We  are 
trying  to  understand  how  you  are  putting  the  picture  together  and 
with  that  I  will  give  you  your  first  piece  of  information  which  is: 
At  1830  a  company  site  force  reinforced  with  tanks  conducted  a 
limited  objective  attack  from  the  northeast  on  the  Leidenhofer  Kopf, 
Mike  Bravo  8617.  Unsuccessful,  the  threat  forces  withdrew  to  hasty 
defenses  vicinity,  Mike  Bravo  888168  and  878180.  OK,  here's  the 
report.  Please  let  us  know  if  you  need  anything  and  we  will  provide 
it  to  you. 

2.  Expert:  It  means  they  attacked,  they  were  driven  off  by  us  and  then 
they  went  to  hasty  defenses.  A  hasty  defense  to  me  implies  that 
they  are  going  to  go  into  either  an  offense  again,  into  an  attack 
mode,  or  they're  going  to  wait  until  reinforcements  come  up,  but 
they're  not  really  going  to  be  in  a  defense  very  long.  Also,  a 
company  size  force  reinforced  with  tanks  means  that  to  me  that  what 
they  were  going  to  do  is  probably  fairly  important. 

3.  Elicitor:  Because  of  the  reinforcement  with  tanks? 

Elicitor  tries  to  determine  the  critical  feature  that  led  to  the 
expert's  conclusion. 

4.  Expert:  Yes.  In  other  words,  they  don't  do  things  also,  where  if 
it's  just  a  company  size  force,  it's  a  straight  motorized  rifle  or 
straight  infantry  force.  That  means  they  probably  think  they  can 
accomplish  whatever  mission  they  were  sent  out  with,  with  the  forces 
they  had  and  it's  probably  not  quite  as  important  as  something  that 
is  tank-reinforced. 
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Military  Consultant:  What  other  thing  might  that  be  though? 

Military  Consultant  looks  for  additional  interpretations . 

Expert:  It  also  might  mean  that  it's  part  of  the  supporting  attack 
and  it's  probably  reporting  an  attack  and  they  wanted  to  have  enough 
forces  there  to  overwhelm  the  friendly  forces.  It  also  implies  that 
they  felt  that  they  didn't  have  enough  firepower  and  enough  of  a 
force  there  to  overwhelm  the  friendly  force  with  their  own  organic 
assets.  The  indicator  is  they  didn't  think  the  company  could  to  it 
by  itself  so  they  attached  tanks  to  it  to  do  that.  It  could  be  one 
of  those  two  things.  It  depends  on  what  kind  of  context  that  you 
look  at. 


Elicitor:  Have  you  made  any  assumptions  based  on  this  piece  of 
information?  We  haven't  told  you  who  you  are  fighting  or  anything 
like  that  or  what  level  you're  at. 

Elicitor  wants  to  know  if  expert  has  generated  any  assumptions  or 
context  given  that  he  has  been  provided  with  none. 

Expert:  I'm  at  least  at  brigade  level. 


Military  Consultant:  Why  do  you  say  that? 

Expert:  Because  they  deal  with  companies  primarily.  I  could  be  at 
battalion,  but  I  doubt  it  because  usually  the  type  of  reporting  I 
get  when  I'm  a  battalion  S-2  I  talk  about  squads  and  platoons.  If 
it's  a  company  size  force,  it  implies  to  me  that  the  battalion  S-2 
is  reporting  to  me  out  of  brigade  and  I  have  this  size  force 
attacking  me  at  this  particular  point,  and  this  is  where  the  center 
mass  of  that  company  is,  but  there's  also  another  implication  here, 
too,  that  reminds  me,  simply  because  you  have  a  six-digit  grid 
doesn't  mean  that  that  company  is  all  scrunched  up  right  there,  that 
they're  on  the  line.  It  could  be  five  hundred  meters  on  either 
side.  That  just  means  that  that  unit  is  center  mass  and  one 
tendency  that  we  have  in  the  Army  is  when  we  teach  people  that  they 
don't  realize  that  there  is  a  lot  of  gaps  but  also  when  someone  says 
something  is  a  certain  location,  that  means  that  unit  is  the  center 
part  of  that  unit  and  it's  spread  out  on  either  side  of  that  center 
point  on  whatever  orientation  they  are  on. 

Elicitor:  If  it's  spread  out  wider  than  normal,  do  they  let  you 
know  that  somehow? 


12.  Expert:  The  only  way  I  can  get  any  indications  of  that  is  to  have 
an  indication  of  what's  on  the  flanks  of  that  unit.  It  depends  on 
where  that  center  mass  is  of  the  flank  units.  The  implication  of 
this  is  I'm  defending  from  the  southwest,  that  means  I've  come  from 
the  southwest  and  they  are  coming  from  the  northeast  which  means 
they  probably  have  a  general  orientation  of  probably  to  the  east  and 
I  have  one  to  the  west.  I  know  it's  in  Germany  so. 


13.  Elicitor:  Why  do  you  know  that? 


14.  Expert:  Because  of  the  Leidenhofer  Kopf.  That's  German. 


15.  Elicitor:  Do  you  know  who  the  enemy  is? 

Elicicor  wants  to  know  if  there  is  an  enemy  associated  with  that 
location. 

16.  Expert:  No,  I  can't  tell. 


17.  Elicitor:  Do  you  have  any  idea  if  you  think  you're  fighting  in 
Germany  and  tanks? 

Elicitor  asks  the  question  again,  using  previous  conclusions  as 
additional  context. 


18.  Expert:  Well,  I  am  assuming  it  is  Soviet  because  we've  been 
conditioned  to  think  it's  Soviet  but  it's  not  necessarily  so.  It's 
probably  Warsaw  Pact,  I  can  say  that  probably.  Unless,  of  course, 
we'd  gone  to  war  with  West  Germans  and  the  French.  But  I  can  assume 
that  it's  a  Warsaw  Pact  force  because  it's  fighting  in  Germany  and 
that's  the  only  type  of  force  we  ever  fight  in  Germany  is  a  Warsaw 
Pact  force.  Now  a  Warsaw  Pact  could  be  one  of  four  countries;  it 
could  be  East  German,  Soviet,  Polish,  or  Czech,  depending  on  the 
location  of  this  particular  fight. 

19.  Elicitor:  Is  there  anything  that  we  could  give  you  that  would  help 
you  make  more  of  this  report  than  your  ability  to  right  now? 

Elicitor  wants  to  see  how  expert  would  proceed  in  his  problem 
solving. 

20.  Expert:  I  need  to  know  what  echelon  I'm  at. 


21.  Elicitor:  What  would  that  do  for  you? 
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22.  Expert.  That  gives  me  an  idea  of  what  size  of  forces  I  need  to  know 
from  that  point.  In  other  words,  if  it's  company  size  forces,  then 
I'm  right  at  the  level  that  my  brigade  commander  needs  to  know.  He 
needs  to  know  the  company  size  forces.  He  needs  to  know  the 
composition  of  those  forces  too.  I  need  to  know  the  composition  of 
the  forces  that  are  on  the  flanks  of  this  unit  and  what  composition 
of  forces  there  are  behind  this  unit,  because  this  implies  and  this 
is  also  another  implication  I  get  from  this--that  is,  since  it's 
reinforced  with  tanks,  it's  an  infantry  unit.  So,  when  I 
graphically  portray  this,  I'll  say  it's  an  infantry  company  plus. 
Because  we  don't  reinforce  tank  companies  with  tanks.  We  reinforce 
tank  companies  with  infantry,  so  it  implies  it's  either...  an  off/on 
thing.  So,  if  you  see  tanks  reinforced  with  tanks,  that  means  that 
this  basic  unit  is  infantry. 


23.  Elicitor:  So  you  need  to  know  what  echelon  you  are  at? 


24.  Expert:  Right.  The  reason  I  say  that  is,  that  way  if  I  know  I'm  at 
brigade  level  then  I  need  to  know,  let’s  say,  you  had  a  company  size 
force  at  this  location  reinforced  with  tanks  doing  a  limited 
objective  attack.  What  happened  on  either  flank? 


25.  Elicitor:  I'll  give  you  that  piece  of  information.  You're  at  the 
division  level. 


26.  Elicitor:  First  of  all,  knowing  that  you're  at  division  level,  what 
does  that  tell  you  now  about  this? 

Elicitor  wants  to  know  how  new  information  helps  expert. 

27.  Expert:  It  tells  me  that  this  is  a  limited  counterattack,  possibly. 
In  order  to  confirm  that  I  need  to  know  what  the  battalion  that  it 
was  assigned  to  is  doing.  Is  it  defending,  or  is  it  on  the  attack? 
My  immediate  thing  is  that  it  was  a  counterattack  because  you're 
talking  about  a  company  size  unit.  At  divisional  level,  I'm 
interested  in  battalions.  That  means  that  a  company  within  that 
battalion  that  I'm  interested  in  is  doing  a  limited  counterattack. 
What  are  the  other  companies  doing  within  it  and  what  are  the 
battalions  on  either  flank  doing  at  the  same  time?  So  what  I  need 
to  know  from  you  almost  immediately  is  what  is  the  enemy  doing,  is 
it  in  the  offense  or  the  defense? 


28.  Elicitor:  I'm  not  going  to  tell  you  that  because  that  is  a  big 

piece  of  information.  We're  only  going  to  give  you  little  pieces  at 
a  time. 
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A  key  feature  of  this  technique  is  that  the  expert  oust  build  his 
context  slowly;  hence,  the  elicitor  does  not  want  to  give  out 
"large"  pieces  of  information. 


29.  Expert:  OK.  That's  the  key  question  that  I'm  trying  to  think  of 
right  now. 

30.  Elicitor:  So  you're  trying  to  put  together  a  picture  in  your  head 
of  what  the  enemy  is  doing. 

Elicitor  tries  to  determine  expert's  objectives  or  problem-solving 
strategy. 

31.  Expert:  Right.  Also,  I'll  tell  you  one  assumption  that  I  can  make 
here  and  I  think  I'm  fairly  confident  about  this  and  that  is  that 
the  company  that  this  belongs  to  is  motorized  rifle  battalion.  The 
reason  that  I  assume  that  is  we  have  a  motorized  rifle  company  that 
is  being  reinforced  with  tanks.  Tank  battalions  don't  do  that. 

They  do  not  reinforce  with  tanks.  So  I  can  make  a  fair  assumption 
that  what  they've  done  is  they  have  taken  tank  company  out  of  a  tank 
battalion,  the  regimental  commander  has  taken  a  tank  company  out  of 
the  tank  battalion,  and  given  it  to  this  motorized  rifle  battalion. 
The  motorized  rifle  battalion  commander  then  has  broken  out  those 
tanks  to  the  company  size  elements.  So  one  of  the  pieces  of 
information  that  I  think  I  already  know  is  that  this  is  a  motorized 
rifle  battalion.  What  I  need  to  know  from  that  point  on  is  I  need 
to  know  if  those  battalions  on  either  side  motorized  rifle  or  tank. 
The  reason  I  need  to  know  that  is  if  I  know  that  there  is  a 
motorized  rifle  battalion  on  one  flank  and  there's  a  tank  battalion 
on  the  other  flank,  then  my  next  question  is  going  to  be,  is  this 
tank  regiment,  are  these  the  first  echelon  battalions  of  this 
motorized  rifle  regiment,  or  is  it  a  tank  regiment? 

In  paragraph  1,  the  data  presented  describe  an  unsuccessful  attack 
by  a  company  sized  force  reinforced  with  tanks.  The  force  then  went  into 
a  hasty  defense.  The  expert  begins  (paragraph  2)  by  restating  the  data. 

He  then  described  a  hasty  defense.  He  lis  .  ;  chree  inferences  from  hasty 
defense --the  enemy  will  go  into  an  attack  m  -V ,  they  will  wait  for 
reinforcements,  and  they  will  not  be  in  a  defense  very  long.  This 
information  can  be  used  to  construct  a  production  rule  with  "hasty 
defense"  as  antecedent  conditions: 


If  enemy  is  in  hasty  defense,  then 

they  will  be  going  into  attack  mode 
or  they  will  wait  for  reinforcements 
and  they  will  not  be  there  very  long. 
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The  knowledge  that  hasty  defenses  are  temporary  ones  can  be  stored 
semantically  by  assigning  the  property  "temporary"  to  "hasty  defense." 


hasty  defense 

|  has  property 
temporary 


This  is  a  good  illustration  as  to  how  production  rules  can  be 
derived  from  structures  such  as  semantic  networks  or  object  frames.  If  an 
entity  has  a  certain  property  or  attribute,  then  inference  rules  can  be 
generated  from  them.  This  can  range  from  simple  rules  such  as  "If 
[entity],  then  [attribute]''  to  inferences  that  can  be  generated  from  those 
attributes  (e.g..  If  defense  is  temporary,  then  units  will  have  moved 
after  X  hours). 

At  the  end  of  paragraph  2,  the  expert  makes  the  inference  that  the 
attack  was  to  do  something  important.  In  paragraph  3,  the  elicitor  asks 
whether  it  is  because  of  the  tanks  that  makes  the  attack  important.  In 
constructing  production  rules,  it  is  necessary  to  specify  what  the 
antecedent  conditions  are.  The  elicitor' s  question  was  directed  at  that. 
The  expert  acknowledges  that  the  tanks  were  the  critical  feature  and  that 
if  the  attack  had  not  been  reinforced,  then  it  would  not  have  been  as 
important  (paragraph  4).  Hence,  we  can  construct  a  production  rule: 


If  enemy  attack  is  reinforced  with  tanks, 
then  objective  of  the  attack  is  important  (2) 


In  paragraph  6 ,  the  expert  elaborates  on  these  by  saying  that  the 
attack  also  could  have  been  part  of  a  supporting  attack.  Also,  the  enemy 
felt  the  attacking  forces  would  not  have  been  strong  enough  to  overwhelm 
the  friendly  force;  hence,  they  were  reinforced.  In  light  of  the 
alternative  interpretation  of  the  reinforcement,  we  modify  the  above 
production  rule  to  include  both  interpretations: 


If  enemy  attack  is  reinforced  with  tanks, 
then  objective  of  the  attack  is  important  (4) 
or  the  attack  is  a  supporting  one  and  more  force 
is  needed  to  defeat  enemy  (6) 

There  is  a  more  general  rule  that  the  expert  discusses- -namely,  that 
when  the  enemy  does  not  have  enough  organic  forces  to  conduct  its  attack, 
it  will  attach  other  forces.  This  gets  represented: 

If  enemy  has  insufficient  organic  force  to 
conduct  a  successful  attack, 

then  it  will  attach  other  forces  to  the  fighting  force  (6) 
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As  noted  in  our  theoretical  discussion  o£  production  rules,  we  can 
use  such  a  rule  to  derive  the  inference  that  if  they  reinforce,  then  they 
thought  the  force  was  not  strong  enough  to  perform  its  attack.  A  useful 
observation  is  worth  making.  The  expert  is  discussing  different  combat 
forces  and  labels  motorized  rifle,  infantry,  and  tanks  as  different  types 
of  forces.  Hence,  we  can  use  this  to  construct  semantic  nets: 


is  a 


Combat  Force 


is  a 


iis  a 


Motorized  Infantry  Tanks 


In  paragraphs  7-10,  the  elicitor  and  expert  discuss  assumptions  the 
expert  is  making  based  on  the  information.  He  states  that  he  assumes 
that  he  is  at  brigade  level  or  higher  since  they  deal  with  companies.  He 
further  elaborates  that  he  would  expect  a  battalion  S-2  to  get  reports  on 
squads  and  platoons.  Finally,  he  assumes  the  battalion  S-2  is  reporting 
to  him  at  brigade  level.  This  information  can  be  used  to  construct 
production/inference  rules  about  who  deals  with  what  level  unit. 


If  report  is  about  companies, 

then  assume  at  brigade  level 

and  assume  report  is  from  battalion  S-2  (10) 

If  person  is  battalion  S-2, 

then  will  get  report  on  squads  and  platoons  (10) 


The  expert  then  (paragraph  10)  says  that  he  assumes  that  the  grid 
coordinates  he  receives  about  the  units  are  the  center  of  mass  of  those 
units  and  that  they  are  spread  out  around  those  coordinates  as  opposed  to 
all  being  located  at  that  particular  point.  He  further  specifies  that  the 
unit  may  be  spread  out  about  500  meters  on  either  side  of  the  grid 
coordinate.  This  can  be  represented  in  the  following  production/inference 
rules . 


If  grid  coordinates  are  given  for  a  unit, 
then  assume  they  ars;  the  center  of  mass  for  that  unit  and 
that  the  unit  may  be  dispersed  around  that  center  (10) 

If  the  unit  being  reported  is  a  company, 
then  it  might  be  spread  out  around  a  500  meter  radius  from 
the  coordinates  reported  (10) 
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Note  that  this  first  rule  Is  actually  a  more  generalized  version  of 
the  second  rule.  This  relationship  can  be  captured  by  noting  that  a 
company  is  a  type  of  unit.  Hence,  by  expressing  this  semantic 
relationship,  the  two  rules  can  be  linked. 

If  grid  coordinates  are  given  for  a  unit, 

then  assume  they  are  the  center  of  mass  /  for  that  unit  and 
that  the  unit  may  be  dispersed  /  around  that 
center  (10)  / 

/  is-a 

If  the  unit  being  reported  is  a  company, 

then  it  might  be  spread  out  around  a  500  meter  radius  from 
the  coordinates  reported  (10) 


In  paragraph  12,  the  expert  responds  to  a  question  concerning  how  to 
determine  the  dispersion  of  units.  His  response  is  to  look  at  adjacent 
units.  Unfortunately,  he  changes  the  subject  without  elaborating  on  how, 
specifically,  looking  at  adjacent  units  will  help  make  this  determination. 
We  can  make  some  guesses,  e.g.,  assume  that  the  enemy  creates  a  gapless 
wall;  hence,  take  the  distance  to  the  next  unit  and  divide  by  two 
(assuming  both  units  are  spread  out  equally).  Unfortunately,  in  our 
determination,  too  many  assumptions  need  to  be  made  on  our  part  in  order 
to  generate  a  detailed  rule  for  this.  At  best  we  can  generate  a  general 
(and  perhaps  useless)  rule. 

If  goal  is  to  determine  degree  of  dispersion  of  enemy  unit, 
then  look  at  flanking  units  (12) 


The  expert  continues  (in  paragraph  12)  to  draw  inferences  to  where 
the  attack  is  coming  from  and  where  he  is  fighting  (in  the  scenario). 

Since  the  enemy  attack  came  from  the  northeast,  he  is  inferring  he  is 
defending  from  the  southwest.  We  can  generate  production  rules  for  this, 
either  for  every  specific  cardinal  point  or  just  a  general  rule  that  if 
the  attack  comes  from  one  direction,  then  the  defense  is  from  the  opposite 
direction.  This  is  a  rather  common- sense  rule.  The  expert  also  assumes 
that  he  is  fighting  in  Germany,  because  of  the  German  name  of  the  enemy's 
objective.  There  is  another  common- sense  rule  which  is  "If  enemy  attacks 
objective  with  German  (or  more  generally,  "country- associated")  name,  then 
assume  battle  is  in  Germany  ("country")."  This  rule  is  probably  not  of 
much  value  since  realistically,  a  member  of  the  G-2  staff  would  never  have 
to  infer  what  country  he  is  in.  Similarly,  it  is  of  limited  use  to  store 
rules  of  the  form,  "If  name  of  something  is  "X,"  then  it  is  part  of  "Y" 
language . " 

In  paragraphs  15-18,  the  elicitor  tries  to  see  if  the  expert  can 
infer  who  the  enemy  is,  based  on  information  the  expert  already  has  (i.e., 
he  is  in  Germany  and  has  fought  tanks).  The  expert  responds  that  he  is 
fighting  a  Warsaw  Pact  country  in  Germany  and  lists  four  possibilities , 
with  the  Soviets  being  the  most  likely  enemy.  This  can  be  represented  by 
a  combination  of  production  rule  and  semantic  network  formats: 
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If 

then 


fighting  in  Germany, 

opponent  is  Warsaw  Pact,  probably  Soviet  (18) 


Soviet  Polish  East  Czechoslovakia 
German 


In  paragraph  19,  the  elicitor  asks  a  question  intrinsic  to  this 
technique- -namely ,  what  additional  information  the  expert  would  like. 

This  is  so  that  he  can  determine  how  the  expert  is  putting  together  the 
situation.  The  expert  replies  that  he  wants  to  know  what  echelon  he  is  at 
(paragraph  20)  since  this  will  tell  him  what  forces  he  needs  to  be 
concerned  with.  He  gives  as  an  example,  that  if  he  is  at  brigade,  then  he 
needs  to  know  about  company  size  forces.  This  can  be  captured  in  the 
following  production  rule. 


If  the  commander  is  at  brigade  level, 

then  he  needs  to  know  about  company  size  forces  (22) 


Actually,  the  expert  switches  back  and  forth  from  talking  about  what 
the  commander  needs  to  know  to  what  the  expert  himself  needs  to  know. 

This  suggests  that  the  rule  should  really  be  more  general  to  include 
intelligence  analysts  (and  possibly  others  as  well) . 

The  expert  discusses  the  need  to  know  the  composition  of  the  forces 
and  the  composition  of  forces  on  the  flanks  and  behind  those  first  forces. 
This  gets  a  little  tricky  because  this  discussion  is  done  in  the  context 
of  a  brigade  commander,  although  it  is  probably  the  case  that  this  rule 
applies  to  all  echelons.  Again,  the  expert  changes  the  topic  before  this 
can  be  pursued;  hence,  we  represent  only  what  he  has  said  and  perhaps  draw 
the  inference  ourselves  and  label  it  in  parentheses  as  we  have  been  doing. 
The  way  to  represent  the  inference  that  other  echelons  care  about 
composition  of  forces,  etc.  is  to  use  a  semantic  net  representation  (which 
is  useful  to  represent  generalizations  with  "isa"  links): 


(If  commander  or  intelligence  analyst  is  at  an  echelon, 
then  he  needs  to  know  about  composition  of  units  /  that  he 
is  fighting  and  composition  of  flanking  /units 

and  composition  of  rear  units)  (22)  / 

J  is-a 

If  commander  or  intelligence  analyst  is  at  brigade, 
then  he  needs  to  know  about  composition  of  units  that  he  is 
fighting  and  composition  of  flanking  units  and 
composition  of  rear  units  (12) 
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The  expert  then  goes  on  to  infer  that  since  the  unit  in  question 
(the  one  conducting  the  attack)  was  reinforced  with  tanks,  then  the  unit 
itself  was  infantry.  He  also  lists  the  rule  that  tanks  get  reinforced 
with  infantry.  Hence,  we  have  two  production  rules: 


If  unit  is  infantry  and  it  is  to  be  reinforced, 

then  it  will  be  reinforced  with  tanks  (22) 


If  unit  is  tanks  and  it  is  to  be  reinforced, 

then  it  will  be  reinforced  with  infantry  (22) . 

In  paragraph  24,  the  expert  gives  an  implication  that  being  at 
brigade  would  have  in  terms  of  what  type  of  information  he  would  need  to 
know;  namely,  that  if  a  company  conducts  a  limited  objective  attack,  then 
he  needs  to  know  what  happened  on  the  flanks  of  that  company.  We  can 
represent  this  by  a  production  rule: 


If  echelon  is  brigade  and  enemy  conducts  a 
company  size,  limited  objective  attack, 

then  brigade  personnel  need  to  know  what  happened  on  the 
flanks  of  that  company  (24) 


It  can  be  assumed  that  there  would  be  a  more  generalized  rule  of 
this  form  which  varies  the  echelon  (e.g.,  division,  battalion,  etc.)  and 
the  size  of  the  enemy  force  that  it  would  be  concerned  with  (e.g., 
battalion,  platoon,  etc.).  However,  the  expert  does  not  give  this 
information  and  it  cannot  really  be  inferred  from  the  text.  This  might  be 
a  question  that  would  have  been  good  for  the  elicitor  to  pose. 

When  the  expert  is  told  that  he  Is  at  division  level  (paragraph  25),  he 
infers  that  the  attack  by  the  company  was  a  limited  counterattack.  He 
states  that  in  order  to  understand  this  better,  he  would  need  to  know  what 
the  battalion  the  company  is  assigned  to  is  doing.  Presumably,  (although 
the  expert  does  not  state  so  specifically),  if  the  battalion  as  a  whole 
were  attacking  then  the  attack  was  for  a  much  larger  objective.  This 
lends  us  to  infer  the  following  rule: 


If  the  enemy  is  conducting  an  attack, 

then  the  size  of  the  attacking  force  is  proportional  to  the 
size  of  the  objective.  (27) 


The  expert  also  states  that  since  he  is  at  division,  he  is  interested  in 
what  the  battalions  are  doing. 

The  expert  also  states  that  since  a  company  within  a  battalion  is 
conducting  the  attack  (instead  of  presumably  the  whole  battalion),  it  is  a 
limited  objective  attack.  We  can  construct  a  rule  from  this: 
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If  a  battalion  uses  only  one  company  to  attack, 
then  it  is  likely  to  be  a  limited  objective  attack.  (27) 


What  is  interesting  here  is  that  there  may  be  a  more  general  rule, 
which  is  if  a  unit  uses  only  a  portion  of  its  subunits  to  attack  then  the 
attack  is  a  limited  objective  attack.  Note  that  this  rule  is  similar  to 
the  above-mentioned  rule  that  states  that  attacking  forces  are 
proportional  to  their  objectives.  This  suggests  two  dimensions  that  can 
be  used  to  infer  enemy  objectives.  Absolute  size  of  force  and  proportion 
of  force.  However,  both  of  these  dimensions  result  from  inferred  rules  so 
we  cannot  use  this  as  a  mode  of  the  expert's  knowledge. 

Some  support  for  this  general  rule  that  we  propose  is  given  by  the 
expert's  request  to  know  what  other  units  are  doing.  Presumably,  if  many 
units  are  attacking  then  the  attack  is  a  major  one  whereas  if  only  the 
company  is  attacking,  then  the  attack  may  be  a  limited  objective  attack. 

In  paragraph  31  (note  that  in  the  previous  paragraphs  the  elicitor 
refused  to  give  the  expert  the  information  he  requested  for  fear  that  it 
might  give  away  too  much  of  the  picture  and  unde mine  the  purpose  of  the 
technique  being  used) ,  the  expert  infers  that  the  attacking  company 
belongs  to  a  motorized  rifle  battalion  since  tank  battalions  do  not 
reinforce  with  tanks.  This  suggests  two  rules:  first,  that  tank 
battalions  do  not  reinforce  with  tanks,  and  second,  [in  this  scenario]  if 
a  combat  battalion  is  not  a  tank  battalion  then  it  Is  a  motorized  rifle 
battalion.  These  two  rules  are  presented  below. 


If  maneuver  unit  is  tank  battalion, 

then  it  will  not  reinforce  its  companies  with  tanks  (31) 


If  enemy  maneuver  battalion  is  not  tank, 
then  it  is  motorized  rifle  (31) 


There  are  some  interesting  points  worth  noting  here.  With  regard  to 
the  first  rule,  there  may  be  a  more  general  rule  that  tank  units  in 
general  do  not  reinforce  their  subunits  with  tanks.  A  question  here  is 
whether  there  are  constraints  on  what  this  subunit  can  be.  For  example,  a 
company  is  one  echelon  lower  than  a  battalion.  Does  this  first  rule  apply 
to  only  those  subunits  one  echelon  below?  Such  information  might  be 
listed  semantically  by  listing  features  of  t\e  units  that  do  not  get 
reinforced.  Another  interesting  question  would  be  whether  an  infantry 
battalion  would  reinforce  a  company  with  infantry,  thus  perhaps  creating  a 
general  rule  that  units  do  not  reinforce  subunits  with  the  same  forces  as 
the  type  of  unit  they  are.  Such  questions  may  have  been  useful  to  pursue 
with  the  expert. 

An  interesting  and  more  general  issue  arises  with  the  rule  that  if 
an  enemy  battalion  is  not  tank  then  it  is  motorized  rifle.  In  addition, 
to  the  question  of  whether  the  rule  is  general  (I.e.,  if  division  not  tank 
then  motorized  rifle,  etc.)  and  also  the  question  of  whether  it  is 
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complete  (i.e.,  in  same  contexts,  there  could  be  airborne  combat  units) 
there  is  also  a  more  fundamental  question  as  to  how  the  expert  arrived  at 
that  conclusion.  Does  he  store  the  two  possibilities  in  the  form  of 
separate  rules?  Another  possibility  is  that  he  knows  that  there  are  two 
types  of  units  so  that  if  it  is  not  one  unit  then  it  must  be  the  other 
unit.  One  way  to  capture  such  knowledge  would  be  through  the  use  of 
semantic  networks  and  "type"  links  as  depicted  below. 


Enemy  Battalion 


type, 

tank  battalion 


type 


motorized  rifle 
battalion 


It  we  focus  on  a  simple  case  with  only  tank  and  motorized  rifle 
units  at  any  level,  we  could  use: 


Again,  such  information  might  have  been  useful  to  pursue  with  the 
expert.  The  expert  gives  a  scenario  as  to  how  the  reinforcement  process 
may  have  proceeded.  Actually,  he  lists  a  set  of  procedures  that  may  have 
been  followed.  These  are  best  captured  by  production  rules  which  can  be 
taken  directly  from  the  text: 


If  goal  is  to  reinforce  subunits  and  unit  is  regiment, 
then  take  tank  company  from  tank  battalion  and  give  it  to 
motorized  rifle  battalion.  (31) 

If  goal  is  to  reinforce  subunits  and  unit  is  motorized 
rifle  battalion  and  has  been  given  tank  company  by 
regimental  commander, 

then  give  tank  platoons  to  motorized  rifle  company.  (31) 


There  are  two  things  to  note  here.  In  the  second  rule,  the  expert 
does  not  explicitly  say  to  give  the  tank  elements  to  a  motorized  rifle 
company.  However,  in  paragraph  22,  the  expert  says  that  only  motorized 
rifle  companies  get  reinforced  with  tanks.  Hence,  we  are  justified  in 
making  this  modification  to  the  rule.  The  second  thing  worth  noting  is 
that  these  rules  form  part  of  a  common  procedure.  The  way  to  tell  this  is 
that  the  antecedents  of  the  second  rule  depend  on  the  consequential 
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conditions  of  the  first  rule  being  met.  Another  consideration  is  that 
there  is  a  common  goal  or  antecedent  condition  to  both  rules.  This  is 
what  is  known  as  chaining  production  rules  to  form  a  larger  procedural 
sequence . 

Having  in^-rred  that  the  unit  the  company  belongs  to  is  a  motorized 
rifle  battalion,  the  expert  wants  to  know  what  the  adjacent  battalions  are 
so  he  can  determine  the  type  of  regiment  that  is  there.  The  first  thing 
this  tells  us  is  something  about  the  hierarchy  of  units,  namely  that 
companies  are  part  of  battalions  which  in  turn  are  part  of  regiments.  We 
can  represent  this  by  using  semantic  networks  and  "has -part"  links: 


Regiment 

|has-part  (31) 
Battalion 

|has-part  (31) 
Company 


Given  this  hierarchy  and  the  fact  that  the  expert  wants  to  infer 
higher-order  units  from  its  parts,  we  can  infer  a  rule  that  one  can  infer 
the  whole  from  its  parts.  This  turns  out  to  be  a  rule  which  experts  used 
frequently  throughout  the  problem.  Such  a  rule  is  more  of  a  general 
problem  solving  strategy  than  a  specific  procedure  or  inference  rule. 

This  demonstrates  that  production  rules  can  be  general  and  high-level  as 
well  as  specific  and  low-level. 

It  is  interesting  at  this  point  to  summarize  how  the  expert's 
problem  solving  process  evolved  using  this  generative  procedure.  A 
remarkable  amount  of  material  has  been  generated  given  that  only  one  piece 
of  data  was  given  initially.  The  expert  began  with  generated  "immediate" 
inferences  from  the  data,  e.g.,  features  of  a  hasty  defense,  why  tanks 
would  be  used  to  reinforce  the  company  that  was  attacking,  etc.  The 
expert  then  went  on  to  make  high-level  inferences  such  as  what  echelon  he 
was  at  and  how  serious  was  the  attack  given  its  size.  Each  round  of 
inference  generated  new  questions  for  the  expert  and  it  was  clear  that 
these  questions  moved  away  from  the  information  he  had  and  more  toward  a 
larger  picture.  For  example,  the  expert  started  forming  hypotheses  about 
the  higher- level  (battalion  and  regiment)  units  he  was  facing  and  became 
more  interested  in  the  adjacent  units  as  well.  It  seems  as  though  the 
flow  of  the  expert's  reasoning  was  to  first  make  inferences  off  the  data 
he  was  given,  infer  immediate  cause  of  the  events  in  the  data  (e.g.,  was 
the  goal  a  limited  objective  attack?),  look  for  a  larger  framework  to  put 
the  events  into  (what  are  the  other  units  doing?),  and  generate  some 
hypotheses  (e.g.,  the  regimental  commander  is  making  a  tank  company 
available  to  the  battalion  commander  for  attack). 

It  appears  that  this  expert,  anyway,  has  a  wealth  of  low-level 
knowledge  he  could  bring  to  bear  on  the  data  he  received  and  was  able  to 
start  forming  the  framework  of  a  larger  picture  based  on  this  knowledge. 
His  knowledge  for  doing  this  scenario  was  largely  representable  in  terms 
of  semantic  networks  and  production/inference  rules,  some  of  which  might 
be  derivable  from  semantic  networks. 
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7.2 


In  Appendix  D  we  present  preliminary  formal  knowledge  models  for 
each  domain  constructed  from  protocols  taken  from  both  interpretative  and 
generative  methodologies.  These  knowledge  models  must  be  considered 
highly  tentative,  since  they  reflect  the  results  of  only  a  limited  amount 
of  time  at  the  early  stages  of  an  elicitation  process.  These  models  are 
constructed  from  those  subjects  in  each  cell  (interpretative/situation 
development,  interpretative/order  of  battle,  generative/situation 
development,  generative/order  of  battle)  whom  we  considered  to  be  most 
expert.  This  determination  was  based  on  three  considerations:  the  type 
and  amount  of  experience  the  subject  had  in  the  relevant  problem  area;  the 
subject's  own  assessment  of  his  expertise  (a  few  subjects  actually 
admitted  that  they  did  not  have  much  experience  in  the  problem  area) ;  and 
the  project  team's  (including  the  Burdeshaw  consultants)  assessments  of 
which  subjects  seemed  to  be  most  expert. 

We  present  data  bases  for  each  of  the  four  cells  described  later  in 
Appendix  D.  A  qualitative  analysis  of  the  databases,  including 
evaluations  of  the  different  elicitation  methods  and  how  they  interact 
with  problem  areas,  in  Section  6.0.  The  data  bases  themselves  were 
constructed  by  assimilating  the  knowledge  representations  constructed  by 
individual  subjects  in  each  cell.  The  inclusion  of  elicited  knowledge  in 
the  database  reflects  that  it  was  presented  by  the  "expert"  and  is  not  an 
indicator  that  it  is  100%  accurate. 
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8.0  SUMMARY 


The  research  described  in  this  report  reflects  an  approach  to 
knowledge  representation  and  expert  knowledge  elicitation  that  is  based  on 
cognitive  theory.  The  goals  of  our  first  year's  work  have  been  to  develop 
methods  that  utilize  either  an  interpretative  (top-down)  or  generative 
(bottom-up)  orientation  to  knowledge  structuring,  and  expert  knowledge 
elicitation,  and  to  investigate  and  compare  the  effectiveness  of 
interpretative  versus  generative  methods  of  knowledge  structuring  and 
elicitation  through  studies  with  army  intelligence  experts. 

Interpretative  knowledge  provides  a  goal-driven  approach  to  specific 
problem-solving,  and  allows  for  formation  of  expectations  and  assumptions 
when  reasoning  under  conditions  of  uncertainty.  Domain  knowledge  of  this 
type  can  be  rather  generalized  or  abstract  in  form  and  may  be  organized 
hierarchically  to  capture  this  type  of  knowledge.  Our  interpretative 
approach  to  knowledge  representation  incorporates  the  use  of  scripts, 
object  frames,  and  mental  models.  The  interpretative  approach  to 
knowledge  elicitation  we  developed  and  tested  involved  semi -structured 
interviews  with  army  tactical  intelligence  experts  which  included  a 
discussion  of  strategies  and  procedures  for  problem-solving  based  on  a 
specific  army  training  exercise. 

Generative  knowledge  is  organized  around  low-level  rules  and 
concepts  in  the  absence  of  a  particular  context.  Problem-solving  with 
this  orientation  revolves  around  individual  pieces  of  data  or 
problem-solving  procedures,  generating  a  framework  by  combining  individual 
pieces.  To  capture  this  type  of  knowledge,  our  generative  approach  to 
knowledge  representation  incorporates  the  use  of  production  rules, 
semantic  nets  and  mental  models .  The  generative  approach  to  knowledge 
elicitation  that  we  developed  and  studied  with  army  intelligence  experts 
involved  a  "data-oriented"  problem  solving.  This  entailed  having  the 
expert  work  on  a  specific  army  training  scenario  in  the  absence  of  the 
background  and  contextual  information  he  would  normally  have  access  to,  by 
being  given  individual  pieces  of  data  from  the  problem  and  being  asked  to 
respond  to  the  data  in  terms  of  what  each  piece  implies  for  the  current 
status  of  the  problem  and  the  direction  of  future  problem  solving  steps. 

8 • 1  Summary  of  Findings 

Findings  from  our  initial  research  studies  with  the  generative  and 
interpretative  methods  that  we  have  developed  will  be  described  in  terms 
of  the  overall  efficiency  of  these  methods  and  the  interaction  of  method 
effectiveness  with  the  characteristics  of  our  test  domains  of  Situation 
Development  and  Order  of  Battle. 

8.1.1  General  observations  on  elicitation  methods.  In  general,  the 
experts  we  interviewed  using  either  an  interpretative  or  generative 
approach  to  elicitation  appeared  to  make  use  of  both  generative  and 
interpretative  knowledge  to  guide  problem  solving,  although  the  reliance 
on  one  sort  of  knowledge  versus  the  other  varied  with  individuals  and 
domains.  For  example,  situation  developers,  as  we  anticipated,  generally 
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appeared  to  make  greater  use  of  generative  knowledge  than  did  order  of 
battle  analysts,  who  typically  do  not  have  to  integrate  as  many  different 
bodies  of  knowledge  to  produce  their  assessment.  Situation  development 
also  had  an  important  interpretative  and  goal-oriented  component,  however, 
since  the  assessment  is  done  to  support  the  commander's  mission  and 
intentions.  The  order  of  battle  analysts  appeared  to  make  extensive  use 
of  their  knowledge  of  various  enemy  force,  doctrine,  and  operational 
templates  which  was  highly  frame -oriented  and  interpretative,  but  they  had 
to  use  a  stream  of  detailed  report  data  to  build  a  specific  OB  assessment, 
and  this  required  a  generative  knowledge  organization. 

These  findings  suggest  that  a  more  efficient  approach  to  knowledge 
elicitation  would  be  one  that  incorporates  both  the  generative  and  the 
interpretative  methods  we  have  developed. 

With  respect  to  elicitation  efficiency,  it  was  also  found  that  all 
of  the  methods  we  studied  were  most  efficient,  in  terms  of  the  quantity 
and  quality  of  knowledge  obtained,  when  used  on  subjects  having  the  most 
skill  and  experience  in  the  target  domain.  For  example,  the  analysis  of 
order  of  battle  interview  data  indicated  that  the  number  of  inferences  and 
hypotheses  that  could  be  generated  from  the  data  was  closely  related  to 
the  subject's  level  of  expertise.  Correspondingly,  the  efficiency  and 
effectiveness  of  methods  seemed  to  decline  in  proportion  to  the  subject's 
apparent  aptitude  for  problem  solving  and/or  level  of  practical  experience 
in  the  target  domain.  This  suggests  that  a  very  careful  screening  of 
potential  experts  should  be  carried  out  in  order  to  identify  the  real 
experts  prior  to  conducting  extensive  interviews;  this  will  maximize  the 
efficiency  of  the  elicitation  methods  that  are  used. 

8.1.2  Elicitation  methods  and  test  domain  characteristics .  The 
following  discussion  summarizes  our  observations  on  the  use  of  our 
interpretative  and  generative  methods  in  the  domains  of  situation 
development  and  order  of  battle  analysis.  Knowledge  used  in  these  task 
domains  was  obtained  in  interview  sessions  that  involved  the  use  of  either 
an  interpretative  or  a  generative  approach  to  elicitation. 

The  analysis  of  protocols  from  the  interpretative  elicitation  of 
situation  development  knowledge  indicates  that  this  approach  captured  a 
lot  of  goals  and  scripts  but  very  little  in  the  way  of  specific 
procedures.  Although  the  interpretative  approach  elicited  few  specific 
procedures  for  situation  development,  the  responses  of  subjects  often 
contained  production  rules,  since  some  of  the  actions  in  their  scripts  are 
situation  specific.  The  interpretative  method  also  revealed  an  extensive 
use  of  organizational  or  systems  knowledge  in  situation  development.  This 
knowledge  enables  the  analyst  to  effectively  coordinate  his  actions  and 
exchange  information  with  other  actors  in  the  intelligence  staff  and  in 
other  functional  areas.  It  also  became  clear  that  there  was  an 
organizational  component  to  the  analysts'  task  that  involved  a  sort  of 
quality  control;  seeing  that  information  is  gathered  and  processed  in  an 
efficient  and  timely  manner. 

The  situation  assessments  produced  (for  the  training  scenario)  in 
the  interpretative  sessions  seemed  rather  general  in  terms  of  the 
conclusions  that  were  drawn  by  the  analyst.  But  this  was  basically 
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consistent  with  the  level  of  knowledge  targeted  by  the  interpretative 
approach  and  with  the  situation  developer's  basic  problem-solving 
orientation.  The  SD  analyst  would  gear  his  reporting  toward  support  of 
his  commander's  mission  and  intent  and  the  commander's  level  of  problem 
focus.  The  analysis  of  data  from  the  interpretative  elicitation  of  order 
of  battle  knowledge  yielded  more  detailed  procedure  descriptions  than 
those  obtained  with  situation  developers.  Since  our  methods  were 
consistent  across  domains,  this  variation  appeared  to  reflect  a  normal  use 
of  more  detailed  and  highly  specified  procedures  in  order  of  battle 
analysis.  To  capture  the  extensive  use  of  these  rather  standardized  and 
detailed  procedures,  the  interpretative  approach  needs  to  provide  a 
rule- like  structure,  or  this  deficiency  could  be  remedied  by  the 
integration  of  the  currently  defined  interpretative  and  generative 
knowledge  structures.  Interpretative  elicitation  for  order  of  battle 
analysis  indicated  that  problem-solving  in  this  domain  involves  relatively 
more  top-down  structured  knowledge  than  situation  development.  Order  of 
battle  analysts,  for  example,  have  a  rich  store  of  frame -oriented 
knowledge  about  enemy  force  structures,  weapons,  and  equipment 
characteristics,  commanders'  personalities  and  so  on. 

The  analysis  of  data  from  situation  development  and  order  of  battle 
elicitation  sessions  using  a  generative  approach  indicates  that  this 
bottom-up  approach  effectively  captures  a  lot  of  rules  of  inference  and 
action.  Components  of  semantic  networks  were  also  obtained,  along  with 
their  linkages  to  rules.  On  the  other  hand,  this  method  of  elicitation 
obtained  very  few  mental  models.  It  seemed  that  the  use  of  such  models 
would  be  fairly  difficult  to  detect  since  their  use  may  be  fairly 
automatic  and  they  may  be  expressed  in  a  "compiled"  (Neves  and  Anderson, 
1981)  form  that  would  be  expressed  very  much  like  a  production  rule.  If 
the  running  of  the  model  has  become  automatic,  then  the  subject  would 
hardly  be  aware  of  its  use.  The  detection  of  such  models,  and  attempts  to 
recover  the  full  and  uncompiled  version  of  the  model,  would  therefore 
depend  on  the  initiative  and  the  level  of  domain  expertise  of  the 
elicitor.  Whether  eliciting  rules,  nets  or  mental  models  it  appears  that 
our  method  should  require  subjects  to  be  even  more  specific  and  detailed 
in  the  description  of  their  knowledge.  The  level  of  description  we  have 
so  far  obtained  is  not  sufficiently  detailed  to  guide  processing  in  a 
knowledge -based  system.  Although  a  large  number  of  rules  were  obtained 
with  our  generative  method,  the  rules  would  often  be  fragmented,  since  the 
bottom-up  method  did  not  address  the  construction  of  a  broader  framework 
in  which  to  place  these  rules.  This  finding  supports  the  need  for  an 
elicitation  method  that  integrates  top-down  with  bottom-up  approaches  to 
knowledge  organization  and  elicitation. 

Throughout  our  interviews,  order  of  battle  analysts  generally 
appeared  to  rely  more  upon  standard  reference  manuals  and  force  templates 
to  accomplish  their  task.  Their  inferences  and  actions  appeared  to  be 
following  army  doctrine  rather  closely.  Situation  developers,  on  the 
other  hand,  tended  to  rely  less  upon  standard  procedures  and  official 
doctrine.  Their  problem  solving  procedures  appeared  to  be  more  dependent 
on  both  their  commander's  particular  mission  and  intentions  and  the 
specific  features  of  the  battle  situation.  The  observations  were 
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consisCent  with  our  Initial  hypothesis  that  situation  development  would  be 
more  generative  and  order  of  battle  would  be  more  interpretative  in  terms 
of  the  direction  and  control  of  problem  solving. 

With  regard  to  the  performance  of  elicitation  methods  across 
individuals,  responses  to  the  interpretative  approach  were  rather 
well-correlated  in  terms  of  the  scripts  and  object  frames  elicited  from 
individual  experts  within  a  domain.  Responses  to  the  generative  approach 
to  elicitation  were  more  varied.  Individual  experts  differed  both  in 
terms  of  what  they  would  infer  from  a  given  piece  of  data  and  how  many 
inferences  a  given  piece  of  data  would  trigger.  When  the  relationship 
between  these  response  variations  and  variations  in  the  level  of  expertise 
is  better  documented  and  better  understood,  this  approach  to  elicitation 
might  have  additional  value  as  a  tool  for  preliminary  screening  of  experts 
for  subsequent  and  more  extensive  elicitation. 

Our  elicitation  studies  in  the  domains  of  situation  development  and 
order  of  battle  analysis  have  also  made  us  more  aware  of  another  important 
and  challenging  dimension  that  must  be  considered  in  the  development  of 
effective  methods  of  elicitation  in  tactical  intelligence.  Both  of  these 
task  domains  exist  to  support  the  problem  solving  of  their  echelon 
commander.  Using  the  terminology  of  cognitive  psychology,  OB  and  SD 
analysts  provide  the  commander  with  an  assessment  of  the  current  problem 
state  and  assessments  of  the  relative  efficiency  and  likely  effectiveness 
of  various  strategies  (i.e.,  friendly  courses  of  action)  for  reaching  the 
goal  state  which  is  the  accomplishment  of  the  commander's  intention. 
Battles  not  only  involve  relatively  static  objects  such  as  terrain  force 
elements  and  equipment,  they  are  fought  across  dimensions  of  space  and 
time.  Indeed,  the  spatial  and  temporal  dimensions  of  battlefield 
assessment  play  a  critical  role  in  the  problem  solving.  Temporal 
considerations  in  a  highly  dynamic  (e.g.,  high  intensity)  combat 
environment  affect  the  information  value  of  the  report  data  that  is 
received:  what  inferences  can  be  made  and  the  level  of  confidence  in 
one's  assessments.  Evaluation  of  the  battlefield  is  based  not  only  on 
inferences  about  the  composition  and  strength  of  forces  but  their 
arrangement  on  the  battlefield.  Different  assumed  configurations  of 
forces  can  have  very  different  implications  for  the  intelligence 
assessment. 

These  features  of  the  tactical  intelligence  domain  offer  special 
challenges  that  are  not  adequately  addressed  with  current  approaches  to 
expert  knowledge  representation  and  elicitation,  and  subsequent  processing 
within  an  expert  system.  The  second  year  of  our  research  will  thus 
include  efforts  to  develop  structures  to  capture  spatial  and  temporal 
dimensions  of  intelligence  experts'  knowledge  and  strategies  for  problem 
solving,  and  the  integration  of  these  sources  of  knowledge  with  other 
knowledge  components.  Another  related  but  broader  representation  and 
elicitation  issue  that  we  will  pursue  in  our  Phase  II  work  concerns 
experts'  formation  and  use  of  inferences  and  assumptions  (including 
spatial  and  temporal  factors  that  introduce  additional  sources  of 
uncertainty)  in  the  course  of  developing  intelligence  estimates. 
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1 . 0  INTRODUCTION 


1.1  Purpose  of  Exercise 

This  report  discusses  an  attempt  at  "microcosm"  research  as  part  of  a 
larger  project  on  knowledge  elicitation  methodology.  Microcosm  research  Is  a 
"bottom-up"  approach  to  generating  and  testing  theory  and  methodology,  which 
focuses  on  specific  instances  of  the  real-world  they  are  to  apply  to,  and 
explores  the  application  of  candidate  ideas . 

In  this  project,  we  are  concerned  with  developing  methodology  to  elicit 
knowledge  which  will  ultimately  be  used  in  particular  situations,  for 
particular  purposes.  A  major  part  of  the  research  effort  is  deductive  in 
spirit  in  that  it  is  to  develop  general  theoretical  principles  yielding 
elicitation  techniques  which  can  be  adapted  to  specific  cases. 

However,  in  parallel,  we  have  begun  pursuing  a  more  inductive  approach, 
working  back  from  specific  instances  in  the  real  world  (or  as  close  to  the 
real  world  as  we  can  get) ,  which  suggest  methodological  (and  even  theoretical) 
insights.  That  is  the  focus  of  microcosm  research.  While  each  approach  can 
be  pursued  independently,  interaction  between  the  two  may  provide  fruitful 
cross-fertilization.  The  inductive  approach  uses  whatever  the  current  state 
of  the  art  in  knowledge  elicitation  technique  is,  which  can  incorporate.  In 
return,  inductive  exercises  can  generate  suggestive  insights  for  the  deductive 
phase.  To  use  an  analogy  from  engineering  design,  we  can  develop  our 
methodology  using  the  principle  of  "build- test-build- test. " 

The  first  step  in  the  inductive  approach,  which  we  report  here,  is  to 
identify  a  single  army  situation  representative  of  a  knowledge  elicitation 
task  to  which  our  tools  of  knowledge  elicitation  would  ultimately  be  applied. 

Specifically,  seven  "knowledge  elicitors"  from  Decision  Science 
Consortium,  Inc.  and  Smart  Systems  Technology,  Inc.  met  with  General  Ennis 
Whitehead,  of  Burdeshaw  Associates  Ltd.  for  about  four  hours  on  the  morning  of 
November  24,  1986.  A  transcript  of  that  session  appears  as  Appendix  A;  and 
comments  keyed  to  specific  places  in  the  transcript  are  given  in  Appendix  B. 
Appendix  C  shows  materials  issued  in  advance  of  the  session- -notably 
guidelines  on  its  conduct;  and  Appendix  D  shows  review  memos  from  four 
participants  at  the  session,  including  a  summary  in  Appendix  D-l. 

Our  working  assumption  at  the  time  was  that  we  were  developing  knowledge 
elicitation  techniques  for  a  particular  purpose,  viz.  eliciting  knowledge  from 
an  experienced  G-2  for  the  purpose  of  enhancing  the  training  of  a  less 
experienced  officer  preparing  to  take  up  the  post  of  Divisional  G-2,  in  the 
early  stages  of  an  east-west  air- land  battle  in  Germany.  The  knowledge 
elicited,  therefore,  did  not  need  to  include  the  knowledge  the  "trainee"  G-2 
had  already  learned  as  part  of  regular  school  training  plus  field  exercises. 
There  was  thus  no  presumption  that  the  knowledge  elicited  would  replicate  the 
expert;  it  would  only  capture  knowledge  incremental  to  that  of  the  trainee. 


This  implies,  effectively,  that  there  would  be  no  scope  for  the 
knowledge  to  be  embodied  in  a  computer  program  to  replace  any  existing 
function  of  a  G- 2  (which  would  have  to  include  the"training  base  knowledge"). 
Indeed,  there  was  also  no  presumption  that  the  knowledge  would  be  represented 
in  any  particular  medium  or  form  (such  as  expository  text  or  computer  code). 

We  were  later  given  to  understand  that  our  primary  focus  should  be 
somewhat  different  from  what  we  had  assumed:  the  domain  of  knowledge  should 
be  intelligence  analysis  more  than  collection;  the  knowledge  to  be  enhanced 
was  junior  to  that  of  the  G-2  (e.g.,  S-2);  the  vehicle  for  communicating  the 
knowledge  should  be  computerized.  Nonetheless,  the  case  illustrated  is  still 
of  some  interest  and  some  of  the  insights  generated  can  be  usefully 
generalized. 

1.2  Format  of  Exercise 

Ennis  Whitehead,  previously  a  G-2  himself,  served  as  the  expert  subject 
and  we  spent  four  hours  with  him  tapping  into  his  expertise  in  intelligence 
collection  management  in  situations  typified  by  a  certain  hypothetical 
scenario,  which  we  dubbed  "the  Treysa  defense  scenario."  It  was  a  scenario 
which  had  been  prepared  by  the  Army  for  the  purpose  of  exercising  the  skills 
of  a  G-3,  and  written  up  at  the  level  of  detail  relevant  there.  General 
Whitehead  was  asked  to  familiarize  himself  with  the  Treysa  defense  scenario 
ahead  of  time,  and  to  give  some  limited  thought  to  what  kind  of  guidance  he 
would  be  in  a  position  to  give  to  a  trainee. 

The  ground- rules  for  the  session,  issued  to  the  subject  and  to  the 
elicitors,  is  given  in  Appendix  C-l  and  the  subject's  initial  statement  of 
relevant  knowledge  is  given  in  Appendix  C-2.  After  the  session,  participants 
were  invited  to  give  their  comments  on  what  they  thought  they  had  learned 
which  was  relevant  to  developing  a  theory  or  methodology  of  knowledge 
elicitation,  or  to  the  direction  of  the  research  project.  Four  of  these  are 
given  in  Appendices  D-l  through  D-4,  of  which  the  first  also  summarizes  the 
content  of  the  session. 

General  Whitehead  was  first  asked  for  any  general  guidance  on 
intelligence  collection  management  that  would  be  helpful  to  the  trainee  G-2, 
which  were  based  on  some  notes  he  had  prepared  (Appendix  C-2).  He  was  then 
asked  to  provide  more  specific  guidance  on  a  priority  intelligence  requirement 
which  he  judged  to  be  of  critical  importance:  uncovering  and  predicting  the 
movements  of  the  enemy's  reserve  forces.  Specifically,  he  was  asked  to  focus 
on  what  he  would  recommend  the  G-2  do  at  0600  the  morning  of  August  19th  in 
the  hypothetical  scenario,  by  way  of  intelligence  collection,  to  determine 
whether  the  enemy's  9th  division  or  7th  tank  army  was  stationary  or  advancing. 

Up  to  this  point,  no  particular  effort  was  made  to  adopt  any  specific 
knowledge  elicitation  technique.  All  questioning  by  the  knowledge  elicitors 
was  unstructured  and  opportunistic.  Both  they  and  expert  were  encouraged  to 
interject  procedural  comments  or  questions  on  the  process  of  eliciting 
knowledge  itself. 


The  latter  half  of  the  morning  was  devoted  to  trying  out  some  more 
structured  elicitation  approaches.  The  first  was  an  essentially  generative 
approach,  "step-through  simulation,"  where  the  expert  responds  to  a  specific 
unfolding  of  the  scenario,  indicating  what  he  would  do  as  G-2  and  commenting 
on  why  (see  Section  3  below) . 

The  second  was  an  interpretative  approach,  involving  top-down 
elaboration  of  goals  from  overall  mission  down  to  tactical  steps  (see  Section 
4.1). 


A  tentative  effort  was  made  to  pursue  another  interpretive  approach, 
that  of  imposing  a  decision  theoretic  structure  on  the  elicitation  (see 
Section  4.2). 

Most  of  the  useful  content  of  this  exercise  is  contained  in  the  comments 
and  reviews  in  Appendices  B  and  D.  Since  they  are  intended  to  be,  at  most, 
suggestive  and  unconstrained  stimuli  to  the  imagination,  no  attempt  has  been 
made  to  seek  closure,  or  even  convergence,  IN  this  first  step  in  a  divergent 
process.  Nevertheless,  we  have  culled  a  few  of  the  more  interesting 
hypotheses  and  insights  generated  below. 
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2.0  OBSERVATIONS  ON  UNSTRUCTURED  PHASE  OF  SESSION 


2.1  On  Knowledge  for  Battle-field  Intelligence  Generally 

o  Elicitation  methodology  cannot  be  usefully  developed  without  a 

clear  understanding  of  the  use  to  which  the  resulting  knowledge  is 
to  be  put. 

o  The  complexity,  variety  and  subtlety  of  knowledge  at  the  G-2  level 
makes  computerized  replacement  of  any  significant  class  of  his 
judgments --whether  of  inference  or  decision- -unpromising. 

Therefore  there  is  little  scope,  or  need  for,  eliciting  knowledge 
in  a  computer-codable  form,  at  that  level. 

o  Although  at  the  outset  of  a  war,  a  G-2's  knowledge  is  driven 

largely  by  the  (interpretive)  doctrine  he  learned  in  training,  he 
expresses  this  knowledge  most  readily  as  instantiations,  i.e., 
even  more  generative  than  production  rules. 

o  However,  the  only  compact,  yet  comprehensive,  way  to  capture  this 
sprawling  instantive  knowledge  may  be  by  imposing  a  formal  logical 
discipline  (e.g.,  decision  theoretic)  on  the  elicitation- -even  if 
it  does  not  fit  naturally  the  subject's  cognitive  style. 

o  The  appropriate  method  of  elicitation  may  depend  on  whether  the 
subject  is  teacher  (interpretive?)  or  a  doer  (generative?). 

2.2  On  Knowledge  for  Intelligence  Collection  Management  (ICM) 

o  Although  Substantive  knowledge  for  decision  making,  such  as  ICM 

is,  may  be  too  situation  specific  to  be  usefully  elicited  ahead  of 
which  it  Is  needed,  however,  procedural  knowledge  on  how  to  make 
such  decisions  may  be  usefully  elicited,  and  perhaps  computerized 
(e.g.  some  version  of  the  "decision  tables"  used  by  some  G-3s  for 
evaluating  major  courses  of  action).  There  may  be  less  scope  for 
transferring  procedural  knowledge  for  the  more  evolutionary  ICM 
decisions . 

o  This  may  be  a  way  of  eliciting  the  knowledge  engineer's  knowledge 
to  enhance  the  judgment  of  the  intended  G-2  "client"  (as  opposed 
to  replicating  the  (substantive)  judgment  of  a  more  expert  G-2). 

2 . 3  On  Knowledge  for  Intelligence  Analysis 

o  Inference  judgment,  such  as  intelligence  analysis  is,  lends  itself 
more  readily  to  using  knowledge  that  can  be  elicited  in  advance 
from  an  expert,  in  being  less  situation-specific. 

o  Procedural  knowledge  that  may  be  worth  eliciting  from  knowledge 
engineers,  to  be  made  available  to  G-2s  or  S-2s  in  the  field  may 
include  aids  for  fusion  of  conflicting  intelligence  reports,  or 
for  eliciting  relevant  judgments  from  staff  colleagues  or 
subordinates  (though  that  knowledge  may  need  to  be  developed  first 
before  it  is  suitably  adapted  to  battlefield  conditions). 
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o 


The  most  promising  form  of  substantive  knowledge  elicitation  may 
be  commonly  recurring  and  relatively  constant  elements  of 
intelligence  analysis,  such  as  the  computation  of  force  ratios  and 
low  level  rules  of  thumb,  that  may  be  taken  straight  out  of 
doctrine  (i.e.,  the  knowledge  is  not  necessarily  elicited  from  a 
single  human  expert) . 
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3.0  A  GENERATIVE  APPROACH:  STEP -THROUGH  SIMULATION 


3.1  whfl&JLa  StsB.-Xhrpvgh  £lffialfl£ign2 

Step- through  simulation  is  an  analytic  technique  but  similar  in  spirit 
to  war  gaming.  It  simulates  possible  sequences  of  acts  and  events  which 
reflect  experts'  judgments  of  their  likelihood  (Ulvila  and  Brown,  1978).  It 
was  developed  primarily  to  generate  a  sampling  of  possible  aftermaths  for  a 
decision  or  situation,  which  is  to  be  evaluated.  The  aftermaths  are  treated 
as  if  they  were  representative  of  the  real  future  conditional  on  the 
situations. 

It  has  many  points  of  similarity  with  conventional  Monte  Carlo 
simulation.  What  distinguishes  it  from  Monte  Carlo  is  that  the  whole  complex 
network  of  possible  sequences  and  their  probabilities  do  not  need  to  be 
pre- specified,  but  are  supplied  by  judgment  as  needed,  as  any  particular  trial 
sequence  unfolds.  This  enables  a  much  richer  future  to  be  explored,  with  much 
less  elicitation  burden  (at  least  for  a  small  number  of  trials). 

Our  intent,  in  this  microcosm  exercise,  was  to  generate  possible  paths 
through  the  future  development  of  the  Treysa  defense  scenario,  with  two  rather 
distinct  objectives  in  mind.  One  was  to  present  General  Whitehead  with  a 
sampling  of  situations  which  would  call  for  G-2  ICM  action,  so  that  he  could 
indicate  how  he  would  respond  to  them  and  have  an  opportunity  to  comment  on 
why.  The  other  was  to  provide  some  basis  for  evaluating  these  choices,  by 
simulating  what  their  aftermath  might  be  (much  as  would  happened  in  learning 
from  experience) . 

In  either  case,  knowledge  elicited  is  in  the  form  of  a  hypothetical 
battle  sequence  (or  set  of  sequences)  that  provide  the  arena  for  a  surrogate 
for  real  experience  involving  apprenticeship  to  an  expert. 

If  the  purpose  of  the  knowledge  elicitation  is,  in  fact,  training  (one 
of  the  purposes  for  which  step  through  was  originally  designed) ,  its  mode  of 
use  would  include  the  trainee  proposing  his  own  solution  at  each  G-2  step, 
before  learning  the  G-2's  solution,  rationale  and  simulated  aftermath.  A 
second  expert  (or  the  first  if  only  one  is  available)  could  comment  on  the 
difference  between  the  trainee's  and  the  expert's  solutions.  Military 
practice  in  training  officers  has  much  in  common  with  this  approach.  For 
example,  submarine  commanders  are  trained  by  participating  in  exercises 
(whether  at  sea  or  simulated  by  computer) ,  after  which  there  are  "hot  wash 
ups"  (and  critiques)  by  experts  who  have  watched  their  performance  In  the 
exercise. 

Of  course,  knowledge  elicited  in  this  form,  only  "stands  in"  for  a  real 
war  sequence,  which  would  be  preferred  if  one  existed  and  were  documented.  A 
distinctive  feature  of  the  knowledge  being  elicited  here  is  that,  although  as 
expert  as  one  can  get,  it  Is  not  grounded  in  any  real  experience  of  the  type 
of  war  situation  in  which  it  is  to  be  used.  Even  experience  from 
participation  in  field  exercises  contains  a  significant  element  of  the 
interpretive  knowledge  held  by  the  exercise  designers.  If  the  real  world  is 
riding  a  bicycle,  and  doctrine  is  a  book  on  cycling,  a  field  exercise  is 
cycling  with  training  wheels. 
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3.2  How  Step  ThrouEh  was  Implemented  Here 

We  asked  the  general  to  place  himself  in  the  context  of  the  Treysa 
Defense  scenario  at  0600  on  August  19,  with  the  intelligence  the  scripted 
scenario  says  he  has  at  that  time.  With  that  information,  he  is  now  asked  to 
construct  a  plausible  sequence  of  friendly,  enemy  and  "natural"  developments, 
starting  at  1000  hours,  given  a  decision  by  the  U.S.  division  commander  to  go 
with  "course  of  action  three"  (on  the  assumption  that  the  commander  has  about 
four  hours  to  make  this  decision  from  0600). 

The  general  was  asked  to  play  three  distinct  roles:  generator  of  the 
real  world  developments  which  the  G-2  is  faced  with  (including  friendly 
actions  other  than  his  own);  his  best  response  to  these  developments  as  a  G-2; 
and  critic  of  those  G-2  actions.  (If  available,  a  different  expert  or  a  set 
of  domain  sp^ific  experts,  would  generate  the  external  developments  that  the 
G-2  is  responding  to;  both  to  make  them  factually  realistic,  and  to  avoid  any 
biases  that  might  creep  in,  due  to  the  fact  that  the  "player"  and  the  "judge" 
roles  are  played  by  the  same  person.  For  example,  there  maybe  a  tendency  for 
a  G-2  to  bias  the  aftermath  of  any  decision  has  has  proposed  in  a  direction 
that  would  vindicate  the  decision) . 

The  general's  instructions  were,  at  each  point  in  time  as  the  sequence 
unfolds,  to:  consider  what  developments  might  happen;  identify  which  is  the 
most  probable;  and  assume  that  it  has  happened.  In  parallel,  he  was  to 
consider  what  his  reaction  as  G-2  would  be  at  each  point  and  discuss  why. 

Only  a  few  steps  in  the  step  through  simulation  were  followed  (see 
transcript  and  comments  in  Appendices  A  and  B) .  In  particular  we  did  not 
proceed  far  enough  along  to  evaluate  sequence  (i.e.,  in  terms  that  would 
enable  one  to  say  whether  the  actions  taken  turned  out  well  or  badly) . 

We  did  not  exploit  a  major  feature  of  step -through  which  is  picking 
developments  in  addition  to  the  most  probable,  in  a  way  that  reflects  their 
likelihoods.  Ideally,  we  would  have  extracted  probabilities  at  each  event 
node  and  sampled  possibility. 

3.3  Implications  for  Future  Use  of  Step  Through 

If  the  purpose  of  step-through  here  were  to  evaluate  a  particular  major 
choice,  (such  as  which  of  three  main  courses  of  action  to  adopt),  by  modeling 
possible  aftermaths  to  it,  we  would  need  a  set  of  aftermaths,  with  their 
evaluated  consequences,  for  each  option,  and  make  it  as  controlled  an 
experiment  as  possible.  Instead,  we  were  primarily  trying  to  generate 
representative  experience  for  a  G-2  for  which  a  single  sequence  may  be 
adequate . 

The  following  would  be  a  more  powerful  variant  of  step- through  worth 
developing  and  testing,  for  the  purpose  of  generating  a  library  of  training 
scripts  for  intelligence  officers,  emphasizing  particular  classes  of  skill  or 
expert  knowledge,  which  they  embody. 
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The  intelligence  function  at  whose  service  the  knowledge  is  to  be  put, 
is  identified,  say  for  intelligence  analysis,  in  keeping  with  the  current 
thrust  of  the  project.  An  example  might  be  the  fusion  of  intelligence  by  an 
S-2  reporting  to  the  G-2.  More  than  one  intelligence  analysis  function  to  be 
aided  may  be  selected,  each  with  its  own  expert  (e.g.,  photo  analyzer). 

Sequences  are  constructed  intended  to  capture  the  range  of  trainee 
actions  be  informed  and  the  aftermath  they  might  lead  to.  Each  uncertainty, 
whose  resolution  constitutes  a  trial  sequence  would  have  its  own  simulation 
expert  e.g.,  on  weather,  own  force  action,  enemy  action.  Critical  commentary 
by  the  same  or  different  experts  could  be  added  at  any  time. 

Various  pieces  of  a  trial  sequence  would  be  developed  by  the  alternation 
of  appropriate  players  in  the  simulation.  The  first  trial  would  be  modal 
i.e.,  each  uncertainty  is  resolved  at  its  most  probable  value.  Subsequent 
trials  would  be  drawn  from  the  whole  range  of  possibilities,  in  such  a  way 
that  selection  is  proportional  to  their  assessed  probability.  In  this  way  the 
resulting  trials  can  be  considered  as  a  random  sample  from  all  possible 
sequences.  (More  efficient  sampling  methods  than  simple  random  can  be 
explored  such  as  periodic  or  Latin  hyper  cube,  but  this  is  a  refinement  that 
need  not  concern  us  initially) .  The  implementation  of  step  through  could  be 
computer  aided  in  the  elicitation  phase. 
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4.0  INTERPRETIVE  APPROACHES 

4.1  Hierarchical  Structure  Approach 

One  of  the  approaches  to  interpretative  knowledge  elicitation  that  was 
tested  during  the  latter  part  of  the  microcosm  exercise  involved  questioning 
that  followed  a  hierarchical  structure.  Ennis  Whitehead  was  first  asked 
questions  about  broad  matters  such  as  ICM  goals  and  plans ,  and  this  was 
followed  up  by  questions  to  "fill  in"  the  more  detailed  levels  of  description, 
reasoning  and  action  that  constitute  these  aggregate -level  objects  and 
processes.  The  subject  was  again  found  to  make  extensive  use  of  specific 
instantiations,  either  to  rederive  some  of  th  broader  concepts  or  to 
illustrate  concepts  for  the  elicitor. 

4.2  Personalized  Decision  Analysis  Model 

4.2.1  Essence  of  approach.  Any  required  judgment,  such  as  a  decision 
or  an  assessment,  can  be  logically  related  to  any  other  relevant  judgments  and 
data  through  a  mathematical  model  based  on  statistical  decision  theory.  In 
fact,  any  sound  reasoned  use  of  knowledge  can  be  expressed  in  terms  of  such  a 
model,  expressed  in  a  standardized  form.  In  particular,  uncertainties  are 
expressed  as  probabilities,  and  value  judgments  or  preferences  as  utilities 
(Watson  and  Buede,  1987). 

This  analytic  technology,  known  as  personalized  decision  analysis  (PDA), 
offers  two  promising  avenues  for  knowledge  elicitation.  One  is  to  discern  a 
logical  framework  underlying  the  expert's  knowledge  and  reasoning  on  any 
particular  issue,  in  order  to  fit  formal  coherent  structure  to  it,  that  comes 
as  close  as  possible  to  the  subject's  existing  patterns  of  thought.  This 
might  be  called  passive  structuring.  Active  structuring  is  where  the 
knowledge  engineer  takes  the  initiative  in  determining  the  PDA  model  and  uses 
experts  (or  any  other  source)  to  determine  the  most  appropriate  inputs. 

There  is  a  marked  practical  difference  between  the  two.  Passive 
structuring  is  essentially  descriptive  of  an  expert;  merely  constraining  him 
to  be  logically  coherent.  In  the  active  case,  any  given  expert  is  only 
providing  part  of  the  input  to  a  prescriptive  model.  No  one  expert  would 
necessarily  concur  with  the  output  and  not  all  of  any  given  expert's  knowledge 
is  necessarily  tapped  (if  it  is  not  in  the  form  corresponding  the  preset 
model).  Some  of  the  knowledge  the  knowledge  engineer  himself  develops  (for 
example  by  analyzing  data  in  ways  that  none  of  the  "experts"  have  done) . 

In  both  cases,  the  expert's  knowledge  must  be  force -fitted  into  a 
language  which  may  not  be  natural  or  comfortable;  the  language  of  probability 
and  utility.  There  are  possible  inherent  ambiguities  of  translation.  There 
is  also  the  problem  of  coherence  and  instability.  What  if,  when  probabilities 
and  utilities  have  been  distilled  from  the  subject  (expert),  they  prove 
inconsistent  with  each  other?  Procedures  for  reconciling  and/or  pooling 
incoherent  judgments  are  currently  being  developed,  (Brown  and  Lindley,  1986). 

PDA  models  would  appear  to  be  examples  of  interpretive  structures  (at 
least  when  specific  prescriptions  or  inferences  are  derived  from  normatives 
rules,  statements  of  goals  as  utilities  and  probabilities). 
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4.2.2  Application  tn  this  case.  A  mild  effort  was  made  to  fit  the 
experts'  judgment  bearing  on  intelligence  collection  management  into  a  PDA 
framework.  There  is  a  well  established  PDA  paradigm  for  evaluating 
information  gathering  decisions  (which  this  is),  known  as  pre-posterior 
analysis  (PPA).  Basically  this  involves  evaluating  each  information  gathering 
option  in  terms  of  the  findings  it  might  generate,  the  actions  these  findings 
might  lead  to,  and  the  probability  and  the  cost  of  erroneous  action,  i.e., 
resulting  from  not  having  perfect  information) . 

Part  of  the  elicitation  session  was  oriented  towards  eliciting  the 
general's  informal  appraisal  of  these  issues  and  to  test  whether  his 
recommendations  appeared  to  be  driven  by  them.  It  did  not  appear  to  be  a 
natural  exercise  for  him  even  in  purely  qualitative  terms,  and  we  did  not 
press  the  matter  on  this  occasion.  We  could  have  pressed  him  harder  to  give 
us  first  qualitative  and  ultimately  quantitative  answers  to  the  appropriate 
questions.  It  seems  unlikely  that  we  would  thereby  be  capturing  the  way  he  in 
fact  formulated  his  conclusions. 

The  focus  of  his  thinking  was  that  the  most  urgent  intelligence  need  is 
to  find  out  what  the  enemy's  reserve  troops  are  doing.  However,  he  did  not 
rationalize  or  verbalize  the  reason  for  this  directly  in  terms  of  the  of  the 
costly  errors  that  the  Army  division  commander  could  make  if  they  misjudged 
this.  He  verbalized  it  in  terms  of  its  being  "the  most  serious  threat."  In 
fact,  at  it's  face  value  this  rule  of  thumb  would  seem  to  indicate  that 
information  about  the  divisions  already  in  contact  would  represent  the 
greatest  threat  and  therefore  merit,  by  this  rule,  the  most  intelligence 
attention.  It  appears  that  his  judgment  conforms  to  what  the  PDA  would  show, 
even  though  the  elements  of  his  thinking  did  not  appear  to  conform  to  that 
model. 


Although  we  were  not,  at  this  time,  looking  for  knowledge  on 
intelligence  analysis  there  were  a  good  many  readings  on  the  subject's 
inferential  processes,  since  these  are  inextricably  bound  up  with  intelligence 
collection  management  decision. 

It  does  not  appear  that  he  thinks  in  Bayesian  updating  terms.  In 
particular,  there  was  nothing  in  his  thinking  aloud  that  appeared  to 
correspond  to  a  "likelihood  function"  (i.e.,  assessment  of  the  diagnostic 
value  of  particular  items  of  information,  expressed  as  conditional 
probabilities  of  observing  the  information,  given  possible  values  of  the 
truth) . 

This  is  not  to  say  that,  in  either  the  ICM  or  the  intelligence  analysis 
case,  there  were  not  perfectly  valid  PDA  models,  which  the  general's  thinking 
reasonably  approximated.  On  any  given  judgment,  whether  of  choice  or 
inference,  there  are  many  perfectly  valid  PDA  models  relating  them  to  more 
primitive  judgments  and/or  data.  A  more  single  minded  effort  than  the 
dilettante  elicitation  session  we  had  on  this  occasion  might  well  yield  one 
that  fitted. 
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ATTACHMENT  A-l:  TRANSCRIPT  OF  MICROCOSM  ELICITATION  EXERCISE  11/24/86 


Subject:  General  Ennis  Whitehead  (BAL) 

Elicitors:  Rex  Brown,  John  Leddo,  Marvin  Cohen, 

Kathy  Laskey,  Theresa  Mullin  (DSC) 

Craig  Cook,  Craig  Will,  Claire  Cardie  (SST) 

(Numbers  in  paren  refer  to  editorial  comments  in  Attachment  A- 2.  Asterisk 
indicates  point  of  special  interest.) 


1.  INTRODUCTION  (1) 

R:  What  I  have  in  mind  is  to  try  to  end  up  with  a  rapid  pass  at  sample 
elicitation. 

Our  metaphor  is:  we're  in  the  business  of  trying  to  design  tools  for  geologic 
excavation.  What  we're  doing  here  is  taking  a  spoon  and  digging  out  a  bit  of 
earth  and  looking  at  it,  that's  the  analogy. 

So  we're  going  to  try  to  get  breadth  rather  than  depth  and  not  seek  any  kind 
of  convergence;  even  if  we  seem  to  be  on  a  roll  and  can  say  "hey,  I  can  see 
how  we  can  really  pick  Ennis’s  brains  thoroughly  on  that."  I  may  divert  us 
just  to  make  sure  that  we've  sampled  all  the  major  types  of  material.  That 
will  be,  as  the  centerpiece  Ennis  is  delivering  (with  some  stimulation  from 
us),  knowledge  that's  representative  of  one  class  of  information  that  we  think 
this  project  is  interested  in. 

We're  going  to  want  two  types  of  material.  First  is  Ennis's  provision  of 
material;  but  In  addition,  any  of  us,  including  Ennis,  that  have  some  side 
comments  that  are  brief.  For  example,  Ennis  may  want  to  say  "Hey,  that's  not 
a  natural  question  for  me.  If  you'd  phrase  it  differently  it  would  help".  So 
I  don't  want  to  discourage  editorial  comments.  We'll  need  to  make  them  brief, 
but  it's  more  useful  to  have  them  made  on  the  spot,  particularly  so  that  when 
the  tape  Is  transcribed  it  Is  a  point  to  think  about  here.  I'd  also  ask  you, 
when  you  contribute,  to  mention  your  name  so  that  the  transcriber  can  keep  us 
apart. 

At  the  end  of  the  session,  we'll  have  a  little  time  for  recap,  but  what  I'd 
like  would  be  for  anybody  who  has  any  comments  that  they'd  like  to  share  to 
write  them  down.  Now  the  comments  can  be  at  all  kinds  of  levels,  including 
"hey,  this  isn't  the  kind  of  knowledge  that  we  should  be  eliciting."  If 
that's  true  then  that's  alright.  Include  also  what  would  be  a  useful  follow 
up  to  that.  But  generally,  give  anything  that,  taking  off  from  this  session, 
you  might  want  John  in  particular  to  dwell  on  as  he  plans  the  rest  of  the 
project.  Concrete. 

We've  asked  Ennis  to  look  at  a  particular  area  of  expertise  which  is  the 
management  of  intelligence  collection.  We've  asked  him  to  take  the  position 
of  somebody  training  a  G-2  that's  getting  ready  to  take  up  a  post  of  G-2  in  a 
situation  like  the  Treysa  Defense  Scenario  (2)  that  we'll  be  using  as  our 
concrete  point. 
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Craig  Cook:  Is  this  training  a  person  who’s  fairly  new  to  the  task,  like  it’s 
their  first  assignment,  or  are  they  a  highly  skilled  G-2  and  they're  just  now 
getting  briefed  on  the  situation  in  this  environment,  having  come  from 
outstanding  performance  someplace  else. 

R:  Good  question.  I  think  it's  a  person  as  inexperienced  as  you  might  expect 
to  be  given  a  G-2  job.  Is  that  a  reasonable  way  to  see  it? 

E:  It's  more  likely  to  be  a  repetitive  position  as  opposed  to  ,  I  mean,  more 

likely  to  be  an  initial  position  as  opposed  to  a  repetitive. 

Craig  Cook:  So  it's  kind  of  like  their  first  assignment  when  they  come  out  of 
G-2  school. 

E:  Yes. 

R:  We've  asked  him,  with  that  in  mind,  to  focus  on  those  kinds  of  knowledge 
which  he  thinks  would  be  terribly  important  for  the  guy  to  have  and  he's  not 
likely  to  pick  up  elsewhere.  So  it's  stuff  that's  sensitive,  or 
controversial,  or  technology  is  advancing  so  fast  it  hasn't  had  time  to  get 
into  the  books,  but  it  will  again  be  Ennis's  judgment  about  what's  top  of  the 

line  for  this  guy  to  know.  Any  questions  on  how  we're  positioning  on  this? 

Initially. . . 

K:  Question.  Would  the  stuff  he's  not  likely  to  pick  up  elsewhere  include 
things  like;  this  section  of  the  book  should  be  emphasized  relative  to  this 
other  section? 

R:  Well,  I  think  that's  fair  game.  It  sounds  like  one  of  those  items  that  if 
the  doctrinal  document  has  been  well  written  somebody  will  have  said  "this  is 
the  most  important  part". 

E:  You've  got  to  know  the  basics  and  the  question  is;  how  much  does  the 
basics  encompass?  That  is  a  hard  question  to  answer.  You  come  up  through  a 
specialty,  but  at  that  point  you've  got  to  be  broader  that  your  specialty. 
Example;  you  will  want  to  use  the  intelligence  resources  of  the  brigades,  the 
aviation  brigades,  the  maneuver  brigades,  the  artillery.  You  don't  control 
them,  you've  never  served  with  them,  you  don't  know  other  than  a  little  bit  in 
school  enough  about  them,  but  you  must  know  them  because  they  are  part  of  the 
resources  that  can  assist  in  the  collection  of  information.  So  you'll  have 
your  own  corps  knowledge  but  then  you  must  have  a  larger  which  is  hard  to 
define. 

R:  Again,  you'll  be  the  judge.  You  have  a  limited  amount  of  time  to  convey 
your  wisdom,  your  sense  of  priority,  what's  near  the  top,  and  that  is  a 
reflection  of  what  they're  most  likely  to  get  wrong,  so  for  example,  if  enemy 
order  of  battle  is  terrible  important,  but  everybody  knows  that,  then  there's 
no  point  to  give  it  precedence. 

Sequencing:  we'll  have  Ennis  first  give  us  any  general  advice,  maybe  5  or  10 
minutes  that's  not  task  specific,  then  pick  out  a  particular  subtask,  and  I 
think  Ennis  has  picked  the  number  one  task  that  Freeman  suggested,  he  gave  a 
list,  and  we'll  start  off  with  the  first  one,  and  just  use  his  judgment  on 
what  he  would  convey.  It's  important  that  the  material  that  we  get  from  Ennis 
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be  able  to  go  into  soae  kind  of  holding  bin  that  could  be  a  written  document, 
conceivably  a  computer  program,  or  even  a  training  program,  that  will  exclude 
direct  tutelage  from  the  expert  to  the  knowledge,  so  that's  a  vehicle  that 
we're  not  including.  Probably  in  this  case  it  will  be  something  that  could 
appear  in  a  document. 

Then  I  will  launch  the  more  aimed  elicitation  with  a  couple  of  little  ideas  I 
have  that  represent  my  view  of  an  interpretive  and  a  generative  approach  and 
then  throw  it  open  for  any  tacks  by  any  of  the  rest  of  you,  (especially  ,  I 
think,  John  and  Marvin  and  Craig,  but  anybody)  that  takes  advantage  of  their 
particular  perspective.  Don't  worry  about  duplication.  I  mean  the  fact  that 
the  issue  has  already  been  covered  when  we  answered  another  question  isn't  a 
reason  not  to  try  it  here,  for  obvious  reasons.  So,  if  you'd  like  to  start  us 
off,  Ennis,  with  your  first  broad  and  then  narrower  contribution. 

J:  Question.  Before  we  begin,  for  the  benefit  of  those  who  may  not  know  it 
should  there  be  an  overview  of  the  scenario  that  we'll  be  using? 

R:  Is  that  something  that  if  you,  for  example,  took  5  minutes,  something 

useful  could  be  got  across?  Why  don't  you  try  it,  and  if  you  can  keep  it  to  5 
minutes  that  would  be  ideal. 

J:  I  probably  could  do  that,  I'm  wondering  if  we'd  be  more  interested  to  have 
General  Whitehead  do  it  ,  I  mean,  that  might  be  a  form  of  knowledge 
elicitation  in  and  of  itself. 

2.  BROAD  GUIDANCE 

R:  Do  you  think?  In  that  case,  why  don't  you  [Ennis]  wait  until  you  get  to 
the  point  of  being  specific.  Here,  give  your  general  comments  and  when  it 
becomes  focused  on  this  [scenario],  take  five  minutes  off  to  set  the  scene. 

E:  I  will  try  to  not  talk  to  everything.  I  see  the  problem  for  a  new  G-2  as 
one  coming  in  prepared,  through  training,  to  the  job,  and  then  to  adapt  what 
he  learned  in  training  to  a  real  live  wartime  situation  and  to  very  quickly 
find  out  where  the  previous  things  taught  that  he  learned  and  experienced  are 
wrong  and  what's  right.  So  he  is  an  inexperienced  G-2,  as  opposed  to  a 
novice.  So  it's  adapting  training  to  the  real  situation  and  I  think  my  notes 
cover  that  (1).  There  is  more  than  knowledge  involved,  and  one  of  the  key 
notes  here  is  that  being  a  G-2  is  not  being  a  one  man  expert,  but 
understanding  how  to  use  people  (2).  So  there  are  some  other  functions  he 
performs  in  addition  to  making  assessments  and  judgments  and  that  sort  of 
thing. 

If  I  were  then  telling  the  G-2  in  this  particular  situation;  "As  I  understand 
it  the  situation  is  laid  out  that  the  enemy  has  been  attacking  fairly 
successfully  without  too  heavy  losses,  we  have  been  delaying  successfully  to 
the  west.  During  the  night,  the  division  commander  pulled  back  a  couple  of 
units  with  the  brigade  headquarters  so  that  he  has  a  very  small  reserve,  two 
brigades  on  line..."  (3) 

R:  Excuse  me.  Ennis  you  might  just  want  to  take  a  moment  to  set  the  scene 
that  it's  a  German  defensive  scenario... 
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E:  This  is  a  typical  Genian  scenario  of  the  fifth  corps  area  called  the  Fulda 
Gap  area.  It's  used  in  all  of  the  Arny  training  to  set  the  stage  to  examine 
issues  and  so  it  makes  a  very  good  one  here.  There  is  sort  of  a  pro-forma  Red 
force  attacking  which  has  a  much  larger  capability  than  the  defending  Blue 
force.  Of  course,  the  purpose  of  the  Blue  force,  in  this  case  the  52  infantry 
division,  is  to  defend  successfully  in  its  sector.  As  the  G-2,  I've  learned 
first  that  on  the  Blue  side  there  are  two  brigades  on  line,  with  most  of  the 
battalions.  There  is  a  small  reserve  with  one  brigade  headquarters.  The 
previous  G-2  has  said  that  the  two  attacking  divisions  have  kind  of  slowed 
down,  they've  run  out  of  supplies.  I  note  that  their  strength  is  still  very 
high.  I  note  that  a  couple  of  regiments  apparently  have  not  been  committed, 
so  I  challenge  that.  All  that  tells  me  is  we  don't  know  enough.  There  is  a 
ninth,  another  large  division,  uncommitted,  behind.  The  report  is  that  it  is 
stationary;  that  surprises  me,  because  it's  within  artillery  range,  1  wouldn't 
think  they'd  put  a  reserve  sitting  still,  within  artillery  range,  when  they 
can  move  it  fifteen  kilometers  back.  So  I  wonder  if  it's  not  attacking  this 
moment  and  we  don't  know  it  (4).  I  realize  that  reporting  takes  one  to  four 
hours  to  get  to  us  at  division  level  so  what  I'm  learning  at  0600  in  the 
morning  may  be  as  old  as  two  o'clock  in  the  morning.  So  it  is  maybe  no  older 
than  one  hour  but  from  the  front  line  back  to  division  does  take  time  to 
report. 

R:  Excuse  me,  Ennis.  Do  I  take  it  there  are  no  broad  messages  that  you'd  want 
to  be  sure  that  the  G-2  leaving  Ft.  Leavenworth  has  in  his  head?  If  your 
judgment  is  that  you'd  want  to  get  into  a  specific  instance  and  say,  "if 
you're  in  a  situation  like  this...",  that's  fine  I  just  wanted  to  check... 

E:  I'm  saying  I  would  not  accept  what  I've  read  in  the  intelligence  summary  as 
being  very  good.  I'm  just  not  certain.  So  when  he  comes  in,  do  not  accept 
the  perceptions  and  assumptions  of  people  who  are  new  in  wartime,  that  they've 
got  it  right. 

R:  So  there's  a  general  message  that's  prompted  by  this  particular  situation. 

E:  Yes.  And  this  being  a  new  wartime  situation  we're  all  new  at  it,  we're 
going  to  our  first  combat  and  we  probably  don't  have  it  right  (5).  So  my 
first  thought  was  when  I  saw  the  situation  is  with  the  new  G-2  he's  got  to 
begin  to  make  a  new  assessment  to,  either  reinforce  that  what  he  has  seen  is 
correct,  or  to  find  out  what  is  wrong.  Then  I  try  to  turn  that  round  into  the 
priority  intelligence  requirements;  which  were  basically:  what's  that  division 
doing  there,  which  could  be;  being  committed  right  now  and  attacking?  What's 
happening  to  the  larger  force  in  the  rear  of  the  seventh  tank  army  which  is 
within  50  kilometers?  (knowing  that  doctrine  is  that  they  hold  them  back 
about  a  hundred  km  until  they  commit  them  as  if  actually  in  motion  about  to 
attack. 

R:  So  the  general  point  that  you  make  out  of  this  is  don't  take  on  trust  the 
intelligence  summary  that  you  receive  when  you  arrive  there  (5). 

E:  And  the  other  point  I'd  make  at  this  time  is  I  think  you're  thinking  in 
too  narrow  a  time  span.  The  G-2  should  be  out  I  would  think  24  to  48  hours 
ahead.  48  may  be  too  far,  but  24  for  sure,  as  opposed  to  looking  at  only 
what's  happening  right  now  which  is  really  history,  maybe  4  hours  old.  So  I'm 
going  to  make  my  thinking  to  get  ahead. 


R:  So  you're  advising  him  to  recapitulate  what's  happened  in  the  recent  past. 

E:  And  to  project  himself  to  get  ahead  of  the  power  curve  in  estimating  the 
enemy's  situation  (6).  The  one  other  point  is  ...1  think  that's  enough  for 
now. 

R:  No,  make  that  point. 

E:  A  perception  is  that  there  is  a  US  corps  and  a  German  corps.  There  is  a 
boundary  between  the  two.  Historically,  they  teach  you  to  attack  on 
boundaries  and,  what's  more,  attack  on  the  boundaries  between  national  forces 
because  their  coordination  between  the  forces  is  less  (7)....  Their  reaction 
to  a  penetration  is  to  protect  their  own  selves,  so  each  side  is  liable  to 
pull  back  on  its  flank  and  allow  you  to  go  on  through.  So  I  said  that  I  would 
send  somebody  up  to  the  23rd  panzer  division  very  quickly,  to  have  an  eyeball 
assessment  of  what  they  think  *s  gonna  happen  (8);  and  make  sure  I  understand 
what's  happening  up  there,  because  the  natural  route  of  that  seventh  tank  army 
is  toward  the  boundary  between  the  US  corps  and  the  German  corps. 

R:  So  you're  mentioning  a  consideration  that  might  lead  you  to  second  guess 
the  intelligence  summary  that  you  received. 

E:  I'm  trying  to  add  to  it.  You're  not  second  guessing;  you're  just  not 
satisfied  with  the  level  of  information  that  you  have  now.  Note  that  I  am 
also  trying  to  get  out  in  front.  I'm  not  trying  to  worry  about  this  battle, 
where  it's  going  on,  but  what  the  larger  forces  can  do  which  has  a  greater 
impact  on  our  division  (9). 

R:  Is  there  a  general  word  of  advice  not  specific  to  this  scenario  but  that 
suggests  to  , . .  for  the  G-2?  Is  there  something  there  that  he  can  carry  away 
(10)? 

E:  I  would  say  you  think  further  ahead  than  the  current  battle.  And  learn 
how  to  use  all  the  intelligence  collection  resources  that  are  available  to  you 
both  directly  under  your  control  and  not  under  your  control.  And  recognize 
that  all  information  that  comes  in  is  based  on  estimates  or  incomplete 
observations  and  that  therefore  you  must  be  prepared  to  challenge  it  and  go 
back  some  other  way  to  get  confirmation  and  yet  you  have  a  responsibility  at 
the  same  time  to  hurry  the  information  into  the  commander.  So  I  guess  the 
other  thing  that  speed  is  more  important  than  accuracy  as  long  as  you're  there 
to  make  judgments  on  what  he's  hearing.  So  get  out  in  front,  have  prompt 
reporting,  and  use  all  the  resources  that  are  available  either  directly  or 
indirectly.  It's  a  management  task  (11). 

3.  SPECIFIC  INTELLIGENCE  AREA:  ASSESSING  ENEMY  MOVEMENTS 

R:  If  that  covers  the  generality  of  this  much  do  you  think  as  is  top  of  the 
line,  would  you  like  now  to  get  into  the  more  specific  task  or. . .or  whatever 
variant  of  it  you  think  would  be  more  helpful? 

E:  The  task  is  the  method  for  tasking  specific  subunits  In  an  intelligence 
assets  to  locate  enemy  regimental,  division  and  army  command  posts  ...  to 


locate  the  headquarters  of  128th  Motorized  Rifle  Division,  the  9th  Tank 
Division  and  the  10th  Combined  Arms...  (1) 

R:  Let's  hold  off  the  second  part  of  that  until  you've  had  a  chance  to  give 
whatever  general  comments  you  feel  would  be  appropriate. 

E:  I  think  of  that  as  being  a  specific  task  but  not  the  task  of  the  G-2.  The 
G- 2  is  saying  ”1  wanna  know  what  those  organizations  are  doing."  A  sub  bit  of 
information  is  where  is  the  CP. 

R:  The  CP? 

E:  The  command  post,  excuse  me.  I  would  much  rather  know  whether  the  ninth 
tank  division  is  moving  or  stationery  than  I  would  to  know  where  it's  command 
post  is.  Now  if  its  command  post  turns  out  to  be  fifty  kilometers  to  the  rear 
then  we've  got  this  data  wrong:  that  division  is  within  fifteen  kilometers  of 
the  front  line  (2) .  But  1  assume  we  have  enough  information  to  know  that  the 
division  is  where  it  is,  so  finding  the  command  post  is  primarily  for  shooting 
at  it  with  artillery  and  that's  a  subtask.  I'm  more  interested  to  know  what 
they're  going  to  do.  So  my  surveillance  which  I  would  put  out:  I  would  ask 
corps  to  give  me  the  mohawk  (which  is  an  aircraft  which  can  see  moving 
vehicles)  I  would,  of  course,  have  it  concentrate  on  one,  looking  in  the  ninth 
tank  division  area  to  see  if  there's  much  movement.  If  you  get  a  lot  of 
vehicle  movement  then  you  can  perhaps  come  to  the  conclusion  that  yes ,  a 
division- size  element  is  moving  and  secondarily  look  deep  toward  the  tank  army 
to  see  if  it's  moving  or  not  rather  than  narrowing  it  down.  I'm  more 
interested  In  what  the  organization  is  doing  as  a  whole,  as  opposed  to  the 
location  of  a  command  post.  The  command  post  helps  to  tell  me  where  it  is  but 
that's  not  a  major  point.  What  intelligence  assets  would  you  use,  the  primary 
ones  that  you  would  have  to  use,  and  this  is  what  a  G-2  should  know  when  he 
arrives.  You  should  ask  for  the  corps  surveillance  aircraft  to  watch  in 
those  areas.  That's  the  mohawk;  they  have  airborne  radar  on  board.  He  would 
ask  his  own  ground  communication  asset,  who  detects  signals  of  radar  (and 
knowing  that  a  division  has  certain  kinds  of  radars  in  it)  if  he  can  detect 
signals  in  this  area,  which  would  be  tied  to  either  the  artillery  or  the  air 
defense  system  of  that  division.  It  would  help  him  to  tell  that  the  division 
is  there.  I  of  course  would  count  on  a  communications  intercept  aircraft 
guardrail,  and  the  ground  sites  which  would  be  listening,  not  only  for  what's 
going  on  in  the  units  or  in  contact,  but  (particularly  guardrail  and  corps) 
you  might  ask  what  they  can  hear  from  there  (3). 

R:  Is  it  helpful  to  comment  on  what  the  underlying  reasons  for  those 
suggestions  would  be  (4)? 

E:  The  threat  of  the  ninth  tank  division  about  to  assault  you. 

R:  So  you  want  to  know  whether  it  is  or  not? 

E:  I'm  trying  to  use  all  these  collectors  to  find  out  what  it's  doing.  Which 
I  think  is  a  more  important  question  than  just  the  location  of  the  command 
post. 

R:  So  in  terms  of  primary  information  requirement,  what  they're  up  to  you 
give  higher  priority  to  where  are  they? 
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E:  Yes. 


R:  Ennis,  is  that  in  your  mind's  eye  tied  to  any  action  implications  for  us 
if  you  find  it  one  way  or  another? 

E:  Yes,  if  they're  crossing  their  line  of  departure  in  the  attack  right  at 
this  moment,  then  maybe  all  three  courses  of  action  don't  make  sense;  because 
I'm  not  sure  the  division  commander  can  implement  a  course  of  action  of 
putting  a  third  brigade  headquarters  in  on  line,  while  a  major  combat's  going 
on  and  rearrange  the  battalions  in  contact.  So  all  the  courses  of  action  may 
be  unfeasible.  It  wasn't  clear  to  me  when  they  were  to  be  implemented,  but 
they  may  be  unfeasible  if  or  impractical  if  the  ninth  tank  division  is 
actually  attacking  at  this  moment.  We  may  know  in  thirty  minutes  which  avenue 
he's  on  so  I'm  trying  to  get  further  ahead. 

R:  As  far  as  a  trainee  G-2  is  concerned,  you  have  any  comments  that  would 
help  him  replicate  the  kind  of  reasoning  that  you've  produced  when  he  doesn't 
have  you  at  his  elbow  (5)? 

E:  The  rule  of  thumb  is  what  is  the  most  serious  threat  to  the  division  and 
to  the  accomplishment  of  the  division  mission?  We  believe  we're  able  to 
contain  the  divisions  which  are  now  attacking  and  in  contact.  We  don't  know 
if  we  can  handle  the  reserve  division,  the  ninth  tank  division,  and  it  is  only 
12  to  15  kilometers  away.  If  it  is  in  fact  sitting  there,  it  is  within  one 
hour  or  less  of  moving,  only  one  hour.  Behind  it  is  a  much  larger  force,  five 
times  our  strength,  or  four  times  our  strength,  which  can  attack  within,  if  it 
continues  its  movement  without  delay,  it  could  attack  in  4  to  8  hours.  So 
those  are  the  greatest  threats,  because  they  can  destroy  the  division  or,  more 
likely,  cause  the  division  to  fail  in  its  defensive  mission. 

R:  And  has  some  clear  implication  for  what  we  should  do  to  head  it  off.  Is 
that  true? 

E:  If  we  know  what  they're  going  to  do  and  where  they  are  then  the  division 
commander,  the  earlier  I  can  tell  him  that  I  have  confidence  in  what  I  think 
is  going  to  happen  the  earlier  he  can  begin  to  adjust  his  forces  to  handle  it. 
Also  I  should  be  thinking  like  the  next  higher  commander  because  he  has  a 
reserve  which  he  can  give  to  me,  to  my  commander,  and  If  I  think  I've  got  a 
problem  and  If  I  think  I'm  going  to  get  the  major  attack  then  I  can  get  a 
commitment  of  that  reserve  brigade  and  I  can  set  up  my  defenses  so  as  to  take 
advantage  of  that  and  make  my  capability  of  either  defending  or 
counter-attacking  more  viable.  And  remember  to  get  that  brigade  there  and  to 
get  these  forces  up  I  need  to  2  to  8  hours  and  if  I  don't  tell  until  the  last 
minute  he  can't  use  them. 

R:  Time  out  for  a  procedural  question.  Is  it  your  feeling  that  the  best  way 
to  get  this  kind  of  skill,  exercise  of  judgment  into  the  G-2's  head  is  to  give 
him  a  concrete  for- instance,  where  you  explain  why  it  is;  as  opposed  to  giving 
general  precepts  of  the  type  that  appear  in  doctrinal  documents  (6)? 

E:  You've  got  to  have  the  precepts,  you've  got  to  have  what  I  call  the  basic 
set  of  knowledge:  training,  education... 
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R:  And  they  have  that. 


E:  And  you've  got  to  teach  that  first.  Then  of  course  I  have  to  comment  on 
the  Army's  system.  What  they  do  now  is  practice  it  in  Leavenworth,  practice 
it  in  the  field  in  command  post  exercises  or  field  training  exercises.  As  a 
more  junior  member  of  the  staff  you  watch  the  more  senior  member  of  the  staff 
do  his  thing,  and  you  do  some  subpiece  of  that  for  him.  So  through  your 
training  and  your  experience  at  a  lower  level  you  then  hopefully  have  the 
basics  to  do  a  good  job  when  your  day  comes  to  be  the  G-2. 

R:  Another  procedural  question.  Having  got  the  for-instance  out,  do  you 
think  it's  helpful  to  make  a  general  comment  or  just  let  the  for- Instance  tell 
its  own  story  without  particular  interpretation?  Would  you  care  to  comment 
one  way  or  the  other? 

E:  I  think  the  general  comment  is  useful.  Because  I  think  the  for- instance 
can  be  misinterpreted  or  has  many, there  may  be  many  general  comments  which 
apply  but  in  my  mind  there'll  be  certain  ones  which  will  appear  to  me  to  be 
more  important. 

CC:  Are  we  in  a  situation  where  you  are  briefing  the  new  G-2  and  you're  going 
to  disappear  and  so  he'd  better  ask  everything  he  needs  to  know  right  now.  Or 
is  he  going  to  take  some  action  about  which  then  he  can  come  to  you  and  say, 
"Here's  what  I've  done,  here's  the  result,  now  what  should  I  do  next?"  or  "How 
should  I  interpret  what  I've  seen  so  far?"  (7) 

E:  The  first.  This  could  be  a  real-life  example.  If  he  could  be  at  the 
corps  headquarters  now  and  the  corps  G-2  says  "You're  going  down  to  the  52nd 
Division  and  here's  what  the  situation  is  and  here's  what  their  latest 
intelligence  summary  says.  Things  are  a  little  different  from  peacetime  and, 
young  fella,  here's  my  advice  to  you." 

CC:  OK.  You  made  a  statement  that  said  earlier  that  you  shouldn't  believe 
the  intelligence  summaries  that  you  receive  out  of  hand,  that  you  ought  to  go 
back  and  re-collect  some  of  that  intelligence  for  your  own  consideration.  My 
question  to  you  is:  "how  do  I  decide  which  sources  to  believe?  How  do  I  even 
know  that  those  troops  are  out  there?"  You  seem  to  take  for  granted  that  they 
were  really  there,  but  you're  not  quite  sure  whether  they're  moving,  or 
something.  Is  there  some  level  at  which  you're  willing  to  say:  "yes,  we 
believe  this  part;"  but  then  there's  another  level  at  which  you're  saying, 
"don’t  believe  that  summary,"  or,  "don't  believe  that  piece  of  intelligence." 
How  do  you  decide  which  to  believe  and  which  not  (8)7 

E:  I  would  say  do  not  not  believe  it.  I  think  you  never  know  enough  to  be 
satisfied.  The  intelligence  summary  is  always  an  estimate,  based  on  limited 
information,  and  people  are  making  judgments  on  it.  You  in  fact  will  begin  to 
make  those  judgments  yourself,  and  could  some  days  be  very  accurate  and  some 
days  not,  based  on  limited  information. 

Now  among  intelligence  collection  resources  (again  part  of  the  basic 
knowledge),  there's  not  much  that  can  see  beyond  a  couple  of  kilometers  or,  to 
use  the  standard  thing,  five  kilometers.  Because  there  aren't  very  many 
collection  means  which  will  provide  beyond  that;  so  in  looking  at  those  two 
deeper  forces  [7th  and  9th]  that  I'm  worried  about,  you  immediately  have  to 
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narrow  yours  to  the  assets  which  can  look  deepest.  Sone  of  then  are  at  a 
higher  level  -  1  mentioned  the  corps  nohawks.  So  you're  going  to  call  the 
corps  and  say  "I  need  your  help  right  now  because  that  doesn't  make  sense  that 
they  be  sitting  that  close,  they  nust  be  noving.  What  are  you  getting  from 
the  surveillance  aircraft?"  (9) 

KL:  I'd  like  to  restate  Craig's  question.  You  looked  at  the  intelligence 
summary  and  said  "Well,  they  say  that  the  ninth  division  is  out  there  and  they 
say  that  the  7th  army  is  back  beyond.  They  appear  to  be  assuming  that  they're 
just  sitting  there.  I  wouldn't  assume  that.  I  would  collect  information  and 
try  to  find  out  whether  they're  just  sitting  there  or  whether  they're  moving." 
But  you  didn't  say,  "Gee,  they  said  the  ninth  tank  division  was  here.  Maybe 
it's  really  not  here.  Maybe  it's  somewhere  else."  So  in  other  words  there 
were  pieces  of  the  intelligence  information  that  you  more  or  less  took  at  face 
value  although  of  course  that  can  be  uncertain  too  but  there  were  certain 
pieces  that  you  decided  to  collect  information.  How  did  you  make  that 
judgment? 

E:  You're  right.  I  made  a  judgment,  rightly  or  wrongly,  that  they  had 
located  the  ninth  tank  division.  In  the  intelligence  business  there  are  some 
sources  that  are  better  than  others.  If  the  source  of  that,  and  I  would  ask, 
"How  do  they  know  that?"  and  I  should  ask,  "How  do  you  know  that?"  If  I  was 
comfortable  with  what  the  sources  were,  then  I  would  more  likely  ....  A  good 
question  is  whether  or  not  it's  there  at  all.  That's  a  fair  question,  and  I 
probably  made  a  hasty  assumption,  assuming  it  was  there,  accepting  the 
intelligence  summary.  Normally  you  don't  put  something  in  that  close  unless 
you've  had  a  pretty  good  indication  (10) . 

CC:  Does  that  mean  that  certain  intelligence  pieces  of  information  are  hard 
data,  that  is  visual  confirmation  or  . . .  direct,  confirmation  and  others  are 
inferred  things  based  on  assumptions,  based  upon  piecing  together  other 
things.  Are  you  saying  that  you  would  rely  upon  the  hard  data  things  and 
question  the  assumption  kinds  of  things? 

E:  Yes.  If  you'd  heard  over  the  radio  somebody  talking  and  your  direction 
finding  put  it  in  that  area  and  you  heard  elements  of  talking  which  appeared 
to  be  in  the  ninth  tank  division  then  you  would  probably  give  that  a  high 
probability  of  that  being  correct.  If  you  also  then  had  a  photo  mission  over 
there  which  showed  lots  of  vehicles  on  the  move  including  artillery  and  tanks 
and  VMPs,  that  would  tell  you  there'd  be  a  million...  (11) 

R:  VMPs? 

E:  That's  armored  personnel  carriers.  That  would  tell  you  there  are  a  mix  of 
forces  which  tend  to  be  in  a  division.  If  they  are  more  than  what  you  would 
find  as  part  of  the  two  divisions,  if  it  appeared  to  be  more  than  just  the 
reserves  of  the  two  divisions  in  contact,  then  that  might  lead  you  to  believe 
yes,  there  is  a  division,  part  of  a  division  there  or  more  than  a  division  or 
something  big  there.  So  you  would  piece  it  together.  I  would  accept  at  face 
value  there  was  some  sort  of  division  in  that  area.  That  may  be  wrong. 

KL:  This  may  be  beating  a  dead  horse,  but  I'd  like  to  push  this  just  a  little 
bit  further.  When  you  looked  at  the  intelligence  summary  you,  immediately 
said,  it  sounds  to  me  like  you  were  saying,  to  yourself,  "If  they're  saying 
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the  ninth  tank  division  is  there,  they  oust  have  this  kind  of  data  and  that 
kind  of  data  and  the  other  kind  of  data,  or  it  is  likely  to  have  come  from 
these  kinds  of  data,  which  are  good."  And  so  when  you  look  at  Intelligence 
summaries  whether,  or  not  they  tell  you  where  they  got  the  data,  you  have 
hypotheses  as  to  where  they  got  the  data  and  whether  they're  likely  to  be  good 
or  bad.  Is  there  any  kind  of  guidance  that  you  could  give  to  the  G-2? 

E:  He  should  go  back  and  check  those,  tfe  have  concentrated  the  subject  on 
what  appeared  to  be  the  greatest  threats  to  the  division  so  there's  a  lot  of 
other  questions  out  there  which  would  have  all  the  same  implications.  But  see 
at  least  what  I'm  doing  (12)  -  I  accept  it  perhaps  too  hastily.  I  don't  know 
but  if  I  ask  for  the  evidence  which  said  it  was  there,  and  felt  comfortable 
that  there  was  a  ninth  division  somewhere  in  this  area,  then  I  want  to  know 
most  importantly  what  it's  going  to  do.  That's  the  part  that  I'm  worried 
about  because  again  it  would  have  the  greatest  impact  on  our  own  forces.  I’m 
driven  by  what  will  have  the  greatest  impact  on  our  own  forces.  And  the  other 
one  we  haven't  mentioned  of  course  are  nuclear  weapons  (13)... 

R:  Is  it  generally  true  that  in  a  broad  class  of  situations  it's  enemy  intent 

and  movement  which  is  the  dominant  concern;  or  does  it  just  happen  to  be  the 
dominant  concern  in  this  particular  scenario  for  reasons  that  are  specific  to 
it  (14)? 

E:  Enemy  intent  and  capability  are  the  dominant  concerns,  because  they  have 
the  capability  of  attacking  and  we're  defending.  The  problem  for  us  is,  "can 
we  conduct  a  successful  defense?"  What  I'm  worried  about  is  will  this  give 
them  a  favorable  force  ratio  which  would  permit  them  to  advance  more  rapidly 
than  they  have  in  the  past  and  therefore  penetrate  our  defense  and  cause  us  to 
fail  in  our  mission. 

R:  With  a  clear  action  implication  for  us  we've  concluded? 

E:  If  I'm  able  to  tell  the  commander  that  they're  sitting  there,  not  moving 

and  don't  appear  to  be  moving,  that  gives  him  more  freedom  of  action  than  if  I 
report  to  him  that  there  actually  seem  to  be  moving  forward  and  we  can  expect 
contact  in  the  next  thirty  minutes  to  an  hour.  So  It  has  an  impact  on  his 
choice  of  courses  of  action  (11) . 

R:  To  look  for  a  contrairy  case,  might  it  sometimes  be  that  it's  obvious  what 
enemy  intent  and  action  is  and  therefore  does  not  deserve  high  priority  in 
additional  intelligence  gathering? 

E:  You  noted  that  I  didn't  put  on  a  high  priority  the  activities  of  the  two 
divisions  which  are  now  attacking.  And  they  are  a  threat.  I  think  they're 
more  of  a  threat  than  the  intelligence  summary  described  but  they  are  less  of 
a  threat  than  that,  so  I  have  sort  of  not  put  the  priority  on  what  they're 
doing.  Again,  part  of  the  education  process.  Those  are  brigade  fights  with 
the  units  in  contact.  I'm  more  Interested  in  their  reserves  and  the  reserves 
behind  them.  The  brigades  are  interested  in  what  they're  In  contact  with  or 
what  they're  about  to  be  in  contact  with.  So  my  problem  is  the  larger  one  of 
the  division.  The  brigade  S2  is  running  the  same  problem  but  at  a  lower 
level. 


A-26 


R:  Tine  out  for  a  procedural  question.  What  we're  doing  now  is  something 
that  one  can  imagine  taking  place  in  a  classroom  at  Ft.  Leavenworth. 

E:  Yes. 

R:  It  sounds  like  a  candidate  way  of  eliciting  information  is  actually  to 
replicate  a  meeting  like  this,  in  the  context  of  a  scenario  with  a  seasoned 
expert,  making  exactly  the  kinds  of  comments  you  are  and  responding  to  the 
kinds  of  questions  you're  getting  from  the  floor  (16).  Any  comment  on  that  as 
being  a  promising. . . 

E:  Yes.  The  only  thing  you  might  want  to  do  is  to  [add  to]  the  elicitation 
experts,  to  add  in,  very  selectively,  a  knowledge  expert  who  can  pick  up  on 
something  that  the  subject  misses  -  that  you  all  might  not  pick  up  -  but  I 
think  this  is  an  excellent  method. 

4 .  OPEN  DISCUSSION 

R:  The  meeting  Is  now  open  for  anybody  to  raise  questions,  such  as  Craig  and 
Kathy  have;  not  yet  testing  out  any  more  systematic  procedures,  but  just  at  a 
common  sense  level,  such  as  one  might  produce  in  a  class  in  a  training  class. 

CC:  How  important  is  it  to  know  what  actions  the  red  forces  could  take;  that 
Is,  how  are  they  limited  by  geography,  weather,  resources,  ammunition, 
whatever,  and  does  one  sort  those  into  some  reasonable  number,  five  possible 
actions  that  they  could  take  or  are  there  hundreds  that  they  might  do?  (1) 

E:  Yes,  you  must  sort  into  courses  of  action.  At  least  it's  normally  done  to 
narrow  them  down  to  a  few  that  you  can  manage.  There  are  three  general 
courses  of  action  here,  and  then  each  general  course  has  its  own  [subactions]. 
For  example  the  divisions  on  line,  which  are  attacking  and  can  continue  to 
attack  the  way  they  are  and  they  do  It  would  have  separate  courses.  The  ninth 
tank  division  could  be  committed  in  the  attack  and  there  are  several  ways  that 
it  could  be  done.  Or  the  the  seventh  tank  army  could  come  marching  along  up 
and  then  it  could  attack  in  the  north  [and  In]  the  south  in  a  completely 
different  corps  area.  So  that's  the  way  I'm  looking  at  it.  There  are  several 
situations,  each  of  which  could  lead  to  a  number  of  courses  of  action. 

CC:  Ok.  Is  it  possible  that  there  are  other  courses  that  you're  not  aware  of 
behind  your  back,  coming  up  from  some  other  direction.  Is  it  wise  to 
concentrate  on  everything,  all  your  resources  on  this  particular  problem  at 
hand;  or  maybe  to  save  some  against  a  surprise  attack? 

E:  You're  always  worried  about  the  things  you  don't  know.  First,  commenting 
on  the  red  conventional  forces .  There  are  enough  of  them  out  there  which 
you've  got  information  on,  that  you  would  feel  fairly  comfortable  that  you 
have  a  pretty  good  knowledge  of  the  general  forces.  What  you  don't  know  are 
use  of  nuclear  weapons,  what  you  don't  know  is  whether  he  has  any  sort  of  deep 
attack  with  airborne  forces,  which  could  interrupt  your  rear  and  those  worry 
you.  As  the  G-2  you  try  and  get  some  sensing  of  the  level  of  that  threat, 
because,  if  you  have  a  threat  in  the  rear,  that  means  the  division  commander 
has  to  put  forces  in  the  rear,  which  makes  him  weaker  up  front.  So  there  is  a 
those  are  important,  and  you  must  make  judgments  on  them. 
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R:  Time  out  for  a  procedural  question.  As  I  hear  you,  most  of  the 
suggestions  you're  coming  up  with  are  not  related  directly  to  what  use  you 
should  make  of  the  information.  It's  focusing  on,  "this  is  information  that 
we  would  like."  My  question  is:  is  this  because  the  way  you  think  about  it 
is:  you  directly  perceive  that  a  kind  of  information  like  "what's  the  enemy 
doing"  is  important  or  is  it  natural  for  you  to  think  "it's  important  because 
it  will  have  a  critical  impact  on  what  I  will  do  thereafter?" 

E:  I'm  thinking  in  terms  of  the  indicators  that  I  can  receive  through  the 
intelligence  means.  I  must  then  make  judgment.  For  example,  with  the  ninth 
tank  division,  if  I  can  see  movement  of  three  large  columns  of  a  hundred 
vehicles,  each  moving  out  of  that  area,  my  judgment  will  be  they  are  about  to 
attack.  That  will  be  the  estimate.  What  I  need  are  the  indicators  from  which 
I  can  make  the  judgments,  so  my  conversation  is  sort  of  concentrated  on  the 
indicators.  If  that  force  in  the  rear,  the  seventh  tank  army  continues  to 
march  westward,  then  it  tells  me  if  it's  getting  closer  and  closer  to  our 
forces  that  means  it  has  probably  finished  its  attack  preparations  and  is 
about  to  attack.  Looking  also  at  the  terrain  gives  me  an  idea  of  which  is  the 
most  likely  direction  in  which  it  will  attack  (2) . 

R:  Presumably,  the  reason  why  you  think  that  information  is  important  is 
because  if  you  thought  about  it  you'd  say,  "hey,  we  could  make  a  costly 
mistake  if  we  guessed  wrong."  My  question  is  whether  thinking  about  the 
costly  mistakes  you  might  make  is  in  the  forefront  of  your  mind  but  you've 
chosen  not  ton  mention  it,  or  whether  you  get  a  direct  sense  for  what's 
important  information  and  it's  only  implicitly  that  you're  figuring  it  out  why 
it  is  that  it's  important.  If  you  agree  that  the  reason  one  information  might 
be  more  important  . . .  than  another  is  that  It  may  save  me  making  a  costly 
error  in  terms  of  subsequent  action  (3). 

E:  No  question  about  it,  this  is  a  two-sided  game.  I  am  worried  about  those 
things  which  appear  to  be  the  greatest  threat  to  the  division  mission. 

Remember  it  comes  back  to  accomplish  the  mission.  In  order  for  the  commander 
to  do  a  good  job  In  doing  the  mission,  he  needs  to  know  what  the  enemy  Is 
going  to  do.  For  example,  ...  threats  to  the  division  first  which  have  the 
greatest  impact  on  a  mission.  Then  technically  I'm  trying  to  give  him  as  good 
an  information,  as  early  as  possible,  so  he'll  have  more  time  to  figure  out 
the  counters.  Then  I  have  to  keep  asking  myself  "Is  what  I  told  him  four 
hours  ago  based  on  the  limited  information,  my  estimate  [now].  I've  got  to 
keep  challenging  my  own  estimate  (4).  So  I  may  go  back  and  say,  "what  did  you 
really  see  out  there?" 

For  example,  when  I  look  at  that  picture  [point  to  map].  One  final  comment. 

It  shows  four  regiments.  That  means,  in  the  lexicon  of  our  training,  that 
they  have  identified  all  four  regiments,  not  the  ninth  division  just  as  a 
symbol,  and  that's  tremendous  information.  Now,  if  they're  not  sure,  they'll 
put  a  question  mark  behind  the  symbol.  Or,  if  they  weren't  sure,  they  would 
just  have  the  division  symbol.  So,  looking  at  the  map,  the  G-2  must  have  very 
good  in  formation  that  the  ninth  tank  division  is  there,  and  very  good 
information  that  it  has  those  four  regiment*  in  it,  and  very  good  information 
that  they  are  generally  where  they  are,  so  I've  accepted  that.  But  I  would  go 
back  and  want  to  know  how  he  got  such  good  information.  So  the  final  point 
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there  is:  soae  training  implied  here,  and  there  is  a  standard  procedure  for 
showing  things  on  maps.  And  that's  part  of  my  estimate. 

Th:  I  have  a  question.  A  lot  of  what  seems  to  be  going  on  when  you  come  up 
with  strategy  for  data  collection  is  role  playing.  You're  playing  the  role, 
putting  yourself  in  the  shoes  of  the  attacking  commander,  and  putting  yourself 
in  the  position  of  you're  coming  through  and  what  he  needs  to  know  and  how 
...in  order  to  get  ahead  of  the  game  in  terms  of  information.  How  much  of  what 
you  have  to  do  is  that  role  playing,  and  how  much  of  it  goes  back  to  looking 
at  what  you  know  about  various  Intelligence  sources,  to  look  at  reliability? 
Do  you  spend  a  lot  of  your  time  role  playing? 

E:  In  theory  I  have  completed  my  training  session  in  the  Army.  I  have  ten 
years  experience,  let's  say.  I  am  now  living  off  that  training  and  education. 
1  know  what  a  mohawk  aircraft  can  do.  I  can  know  what  a  ground  communications 
collection  system  can  do.  I  know  what  a  radar  ....  can  do.  So  I  now  have 
moved  up  to  a  higher  level  of  trying  to  put  it  all  together  (4) . 

? :  How  do  you  know  how  to  play  the  roles  of  your  commander  and  the  other 
commanders? 

E:  I'm  probably  not  playing  the  role  of  my  commander,  because  he  probably 
asked  me  these  questions.  It  could  be,  I  hope  I  anticipated,  he  may  have 
already  asked  them.  On  the  other  side  again,  based  on  my  training,  I  have 
gone  through  the  school  systems  and  I  have  learned  what  the  Soviet  doctrine 
is,  have  learned  how  they're  organized.  And  what  I  am  trying  to  do  now,  so  I 
am  role-playing  what  I  learned  in  peacetime.  What  I  should  be  sensitive  to  is 
how  much  of  that  is  wrong.  Or  what  is  different.  But  the  doctrine  in 
peacetime  tells  me  that  they  keep  their  reserves  farther  back  outside  of 
artillery  range,  outside  of  surveillance  range  and  then  they  move  them  forward 
very  quickly  into  the  attack.  This  doesn't  fit  that  model  so  I'm  challenging 
the  sitting  still.  I  think  it  must  be  attacking.  I  could  be  wrong,  but  I  did 
put  it  against  the  model  of  what  I  learned  about  the  doctrine. 

CC:  Now  does  part  of  what  you're  telling  the  new  G-2  include  an  assessment  of 
what  the  commander  prefers,  what  style  of  intelligence  information  he  likes, 
what  his  conservative  versus  aggressive  kind  of  behavior  is,  what  is  a  .... 
story  to  tell  him  versus  a  story  that's  not  useful?  Is  it  commander* specific 
as  well  as  just  the  ...  (5)? 

E:  I  probably  have  a  commander  model  in  my  head  I  don't  recognize,  I  can’t 
identify  right  now.  I  do  know  that  some  commanders  will  claim  they  are  better 
G-2s  than  the  G-2  which  means  that  he  will  play  a  very  active  ...  in  saying 
"why  do  you  know  this?  Do  this,  do  that."  He  will  be  a  very  strong  leader 
...  Other  division  commanders  may  be  more  passive. 

R:  Is  the  expert's  contribution  where  to  put  the  priorities?  Is  it, 
alternatively,  that  he'll  be  in  a  better  position  to  implement  the  search  for 
a  particular  type  of  information?  Is  it  a  blend  of  the  two  or  something 
different? 

E:  I  would  think  with  his  training  everybody  always  thinks  they  can  give  good 
advice.  My  perception  was  to  give  him  three  things  (6):  one,  tell  him  to 
concentrate  on  the  high  priority  things,  in  other  words  prioritize.  You  can 
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be  caught  up  in  a  division  comaand  post  or  corps  command  post,  you  can  be 
caught  up  in  the  terrible  details.  So  1  guess  one  of  the  things  is  to  keep 
your  eye  on  the  high  priority  things.  Look  in  great  depth  and  detail  to  see 
if  the  people  who  are  giving  the  information  know  what  they're  doing.  But 
that's  a  spot  check.  Keep  your  eye  on  the  high  priority. 

The  second  point  again  is  to  learn  how  to  use  all  of  the  resources  available 
to  you  which  are  directly  available  or  indirectly  and  take  full  advantage  of 
them.  In  other  words,  get  110%  out  of  100%  of  . . . 

R:  Let  me  now  sharpen  the  question  a  little  bit.  Do  you  have  any  concerns 
that  . . .  the  G-2  that  only  had  the  standard  training  and  read  the  standard 
documents  will  have  a  misapprehension?  For  example,  are  you  responding  to  a 
risk  that  he  may  just  go  through  the  list  of  informations  routinely  called 
for,  without  taking  the  particulars  of  this  situation  into  account.  So  I'm 
asking  you  whether  you  see  any  important  gaps  in  your  typical  G-2  as  he  comes 
out  of  Ft.  Leavenworth  that  your  comments  may  help  to  plug?  Does  he  need  the 
information  we're  eliciting  or  he  probably  could  manage  just  fine  without 
you're  ministrations  (7)? 

E:  I  don't  know  the  answer  to  that  question.  I  really  don't. 

R:  It  prompts  a  more  general  question,  really  underlying  the  whole  project, 
which  is:  how  important  is  it  to  elicit  knowledge  that  goes  beyond  what  people 
acquire  anyway?  And  if  you  could  comment  on  it,  focus  on  this  case,  if  you 
find  that  convenient? 

E:  I  intuitively  believe  that  there  is.  I  cannot  factually  assure  you  that 

it  Is  true. 

R:  Is  there  anything  for  example  that  you'd  be  concerned  the  G-2  might  do  If 
you  hadn't  got  to  him? 

E:  I  start  from  a  baseline  that  all  everyone  only  knows  a  portion  of  their 
job.  No  one  ever  knows  it  fully  and  in  wartime  you're  experiencing  it  for  the 
first  time.  I  have  watched  people  perform  in  combat  and  those  who  receive 
good  training  and  education,  not  just  from  the  school  system,  but  models  set 
by  other  people,  do  better  in  combat  quite  often  than  those  who  have  not.  So 
I'm  saying  that  the  experience  based  on  that  sort  of  observation,  an 
experienced  G-2  or  an  experienced  expert  could  provide  some  value  added  to  the 
training  which  it  provided  in  the  school  system.  As  an  intuitive  statement. . 

Th:  Question.  One  of  the  things  that  you  did  when  you  went  through 
instructing  the  trainee  G-2  besides  mentioning  prioritizing  collection  and 
using  all  resources  was  at  least  I  thought  you  were  doing  this,  identifying 
sources  of  uncertainty  and  identifying  places  where  the  patterns  you  might 
expect  or  recognize  from  your  training  might  be  wrong.  So  one  of  the  other 
things  I  thought  I  saw  going  on  was  an  emphasis  on  sources  of  uncertainty  that 
you  might  not  anticipate  from  your  training,  that  the  real  world  may  have 
uncertainties  and  unexpected  things  (8) . 

E:  Yeah.  From  the  training  I  find  that  the  Soviet  division  is  in  a  strange 
location  to  be  sitting.  So  I  assume  it  is  attacking.  But  I've  also  got  to 
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find  out  whether  my  perceptions  or  the  training  is  right.  I'll  also  learn 
whether  it's  right  or  not,  when  I  find  out  the  answer  to  the  question. 

Th:  Question  again.  Is  it  important  for  a  G-2  to  realize  that  their  training 
may  be  flawed  and  to  be  aware  of  that  at  all  times?  Is  that  something  else 
you're  trying  to  teach  them,  be  aware  that  your  training  has  limited 
applications  in  this  real  battle. 

E:  I  guess  I  would  use  the  term  the  training  is  incomplete.  It  could  well  be 
flawed.  It's  surely  incomplete. 

M:  We  have  a  case  here  of  one  type  of  knowledge,  a  knowledge  of  doctrine 
conflicting  with,  presumably,  evidence,  intelligence  summary  suggested, 
whoever  prepared  it,  that  this  division  was  included  [?].  And  you're 
suggesting  let's  find  out  if  it's  true.  I'm  interested  in  two  things.  One 
is,  how  you  weigh  these  competing  sources.  In  other  words,  what  if  we  go  back 
and  check,  and  still  aren't  real  sure?  The  second  part  of  my  question  has  to 
do  with  the  fact  that  if  they  were  moving  that  would  have  serious 
consequences.  What  if  it  were  reversed  and  the  estimate  here  was  that  they 
were  moving  and  doctrine  suggested  they  weren't?  In  other  words,  the  summary 
that  you  read  when  you  came  into  the  job  said,  "yeah,  they're  attacking,"  and 
for  some  reason,  doctrine  suggested  to  you  that's  very  unlikely.  Would  you 
then  not  bother  because  the  estimate  was  suggesting  that  situation  and  your 
prior  knowledge  suggests  the  situation  might  be  better  than  the  estimate 
rather  than  worse  . . .  Would  you  be  equally  enthusiastic  to  employ  intelligence 
resources  to  double  check  the  estimate  if  they  were  going  with  the  worst  case? 
Is  It  because  they're  going  with  the  best  case  that  you’re  more  motivated  to 
double -check? 

E:  The  reason  I  had  concentrated  on  those  two  forces  is  because  maybe,  as  a 
G-2,  I'm  taking  the  worst  case.  What  could  do  the  most  damage  to  my  division 
and  its  mission?  And  those  two  forces  can  do  the  most  damage.  And,  since  it 
the  one  division  Is  only  ten  kilometers  away,  I  need  to  know  what  it  is  doing. 
I'll  be  very  happy  if  he's  moving  away  from  me;  I’ll  be  fairly  happy  if  he's 
stationery;  I'll  be  more  agitated  if  he's  moving  forward,  because  the 
commander  will  be  agitated.  So  again,  regardless  of  what  it's  doing,  it  is  a 
significant  threat  to  the  division  mission.  So  remember  in  my  job,  I'm  taking 
everything  from  the  division  mission.  Which  gets  us  back  in  that  old  short 
term  [priority]  which  is:  mission,  enemy  situation,  terrain,  own  troops  and 
time.  The  first  one  is  mission  (9). 

M:  Let  me  put  a  question  specifically.  What  if  you  instruct  the  mohawks  to 
look,  but  because  of  cloud  cover,  or  whatever,  the  results  are  Inconclusive? 

E:  They  get  shot  down. 

M:  Then  what  goes  into  your  estimate? 

E:  In  my  list  of  collectors,  I've  got  as  many  as  ...  For  example,  this  would 
be  very  high  priority,  both  to  the  division  and  to  the  corps,  to  have  an  Air 
Force  give  this  coverage  also.  So  you  would  probably  not  only  use  your  own 
division  intelligence  collection  resources  that  could  reach  out  there,  and  ask 
for  the  corps  to  give  you  coverage,  but  in  this  case,  because  they  are  such  an 
important  threat,  you  would  ask  for  Air  Force  reconnaissance. 
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M:  Let's  assume  that  if  there  were  no  [special  U.S.  action?},  they  would  be 
here  in  a  half-hour,  and  there's  nobody  that  can  really  verify  it  in  that 
time.  You  have  to  give  some  input  to  your  commander  right  now  (10). 

E:  There  are  some  other  things  that  a  G-2  would  do.  You  would  ask  the 
brigades  if  they  had  any  signs  of  reinforcements  (11).  You  would  ask  the 
aviation  brigade,  which  has  a  lot  of  helicopters,  are  they  seeing  any  sort  of 
movement  back  there  visually;  because  if  it's  a  clear  day  they  may  be  able  to 
see  five,  six  or  seven  kilometers  deep.  You  would  ask  the  artillery  if  their 
airborne  observers  are  seeing  anything.  You  would  go  to  your  signals 
collection  managers  and  see  if  they're  getting  any  new  signals  and  locations 
of  signals  along  the  march.  You  would  probably  put  your  EH60  helicopters  up, 
which  can  collect  radar  signals,  to  see  if,  as  they  moved  on  the  march  and  had 
their  air  defense  with  them,  are  their  air  defense  radars  moving  along  there? 
All  these  would  be  indicators  that  maybe  something  [is  going  on].  You're 
worried  enough  about  that,  because  it  is  a  threat.  You  would  like  to  find  out 
for  sure  what  it's  doing  and  you  would  concentrate  your  collection  of 
resources,  those  that  can  see  over  the  hills,  to  try  and  give  you  more 
information. 

CC:  To  follow  up  Marvin's  question,  if  you  cannot  determine  conclusively  what 
it  is  that  they're  doing,  how  do  you  express  this  uncertainty  back  to  the 
commander,  to  kind  of  convey  that  you  haven't  been  able  to  reach  a  conclusive 
choice,  but  he  needs  to  take  some  action  soon  anyway? 

E:  My  conclusion  to  the  commander  would  be,  if  2  or  3  hours  went  by  and  I 
had' t  gotten  some  feedback  from  the  collectors  that  I've  asked,  or  the  people 
who  are  analyzing  collection  to  look  for  any  signs  that  make  sense,  I'd  say  my 
conclusion  would  be  that  they're  stationary,  because  I  have  seen  no 
indications  of  moving.  And  I  believe  I  could  see  indications  of  moving. 

CC:  So  you  would  draw  a  conclusion,  you  would  not  say,  "Boss,  I  don't  know 
what  the  answer  is.  All  of  our  sources  are  down.  We  just  can't  tell."  You 
would  draw  some  kind  of  conclusion  based  on  whatever  evidence  you  have  arrived 
at. 


E:  And  I  would  tell  him  what  that  evidence  was.  We  have  seen  no  evidence  of 
heavy  movements  to  the  west  of  that  area.  We  have  seen  no  new  signals  of  new 
communications  net  which  indicate  a  force  moving  up.  We  have  seen  no  radar 
signals  which  would  indicate  that  the  sands  [?]  are  moving  forward.  We  have 
seen  no  new  artillery  in  position  firing  from  our  artillery  observers.  I 
would  give  him  all  the  negatives.  So  the  best  I  can  tell  him  is  the  ninth 
tank  division  remains  there,  hopefully,  or  I  begin  to  see  evidence  that  it's 
not  there. 

R:  Can  you  give  a  measure  of  uncertainty? 

E:  You  can  give  a  measure  of  uncertainty. 

5.  STRUCTURED  ELICITATION:  STEP-THROUGH  SIMULATION 

R:  Even  though  we  clearly  have  not  exhausted  what  Ennis  has  to  say  that's 
useful,  I  think  now's  the  time  for  us  to  try  out  some  more  structured 
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approaches  -  Che  kind  of  things  Chat  sight  not  cone  out  of  a  classroom  bull 
session.  1  have  two  in  mind,  but  1  don't  want  to  push  those  ahead  of  any 
other  systematic  approaches  anybody  else  wants  to  try. 

The  first  one  I'd  like  to  try  is  what  I  call  step-through  simulation  which  is 
very  similar  to  traditional  war  gaming  (1).  Normally  you'd  have  perhaps 
several  people  playing  different  parts  of  it.  Today  we're  going  to  try  and 
just  pick  your  brains,  since  that's  the  name  of  this  particular  game.  What 
I'd  ask  you  to  do,  in  the  same  context,  is  to  draw  out  a  sequence  of  events 
from  your  imagination.  Tell  us  if  that's  too  much  of  a  challenge  but  if  you 
wouldn't  mind  trying,  please  follow  two  different  strands.  One  is  the  world 
unfolding  as  the  G-2  sees  it,  and  that  includes  what  the  environment  is  doing, 
weather,  what  the  enemy  is  doing,  and  what  your  colleagues  are  doing.  Those 
ones  you  can't  control.  And  then  the  other  strand  is:  what  you  would  be 
thinking  about  as  a  G-2  and  in  particular  what  you  would  do  as  this  imaginary 
scenario  unfolds.  It's  going  to  be  probably  a  little  bit  difficult  for  you  to 
both  play  the  world  and  play  the  G-2  but  we  don't  have  two  characters.  But 
you  might  like  to  comment  on  the  contamination  -  effect  not  just  the  burden 
effect,  but  the  contamination  effect  of  having  to  play  both  sides  of  that.  Is 
that  reasonable,  to  ask  you  to  take  one  step  at  a  time,  starting  at  ,  well, 
0600  on  the  19th  or  a  different  more  convenient  point  in  time? 

Let  me  give  you  a  little  prompt  if  you  find  this  helpful,  but  I  don't  want  to 
steer  you  when  you  have  an  impetus  of  your  own.  If  you  could  take  the 
position  that  the  commander  of  the  division  is  basically  intent  on  one  of  the 
courses  of  action,  say,  course  of  action  2.  However,  we  understand  that,  as 
events  unfold  the  commander  may  change  his  mind.  You  may  start  off  in  that 
direction.  Assume  that  you  know  only  what  the  G-2  might  learn.  He,  of 
course,  wouldn't  learn  for  sure  what  the  enemy's  doing.  More  realistically, 
you  would  not  make  any  assumptions  that  the  G-2  wouldn't  know.  It's  probably 
best  not  to  make  any  assumptions  that  we  know  something  (from  the 
scenario -writer)  that  the  G-2  wouldn't  know. 

J:  General,  let  me  interject,  why  don't  we  go  with  course  of  action  3,  the 
balanced  thin  line  that  you  liked  with  the  large  reserve  maybe  forward,  as  you 
would  have  placed  it. 

E:  Would  you  do  one  other  thing,  would  you  give  me  a  time  frame  or  do  you 
want  me  to  invent  it,  when  it's  supposed  to  be  implemented? 

R:  Let's  try  this  on  for  size:  we  take  the  scenario  as  described  here,  which 
as  I  understand  it  is  that  you're  at  0600  and  you've  got  about  4  hours  to 
decide  on  a  course  of  action.  Unless  you  have  a  different  suggestion,  suppose 
the  4  hours  is  over,  so  we're  at  1000  and  the  commander  has  decided  on,  as 
John  suggests,  course  of  action  3,  and  now  the  camera's  rolling. 

E:  Well,  let  me  ask  one  more  question.  He  decided  on  course  of  action  3  at 
1000  hours,  and  when  is  it  to  be  implemented?  Is  it  during  that  same  daylight 
hours  or  is  it  to  be  done  the  next  night  or  . . .  let's  make  it  as  soon  as 
possible. 

R:  I  think  we  should  use  your  judgment,  whatever  seems  a  plausible,  most 

likely  sequence.  Whatever  your  imagination  throws  at  you  as  being  a  plausible 
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unfolding  of  the  scenario.  And  don't  worry  about  it's  being  the  most 
plausible  but  just  a  plausible  one. 

E:  The  larger  situation  which  I  am  imagining  is  that  at  10:00  the  division 
commander  has  set  down  the  new  course  of  action  3  and  will  Implement  it  now. 
Basically  what  that  does,  from  the  maneuver  forces  side,  is  that  he  will  begin 
to  withdraw  some  of  the  battalion  task  forces,  which  are  now  forward,  and  to 
move  the  brigade,  the  small  brigade  which  is  in  the  rear,  to  another  location 
and  to  build  that  reserve  up  larger.  So  he  will  continue  to  fight  with  2 
brigades  forward,  the  same  organization  he  had;  but  he  will  take  away  some  of 
their  forces  to  build  a  bigger  reserve,  a  fairly  logical  sequence  of  events. 
He’s  going  to  do  it  now  during  daylight  hours,  because  he’s  assumed  that  he 
can  move  on  the  road  without  too  much  harassment  from  the  air  and  that  he  can 
extract  those  battalions  which  are  forward  and  get  them  out  of  contact.  It’s 
doable . 

R:  Is  there  anything  that  as  G-2,  you  would  put  in  motion  right  away? 

E:  Two  time  frames.  It  is  too  late  to  do  much  right  now  for  something  which 
is  going  to  happen  starting  now.  I  come  back  to  my  original  point,  I've  gotta 
try  and  get  12  to  24  hours  out  ahead.  Having  said  that,  I  still  must  do 
everything  possible  to  see  if  there  is  going  to  be  a  major  threat  which  will 
prevent  that  course  of  action  from  being  carried  out.  The  course  of  action 
being  to  thin  out  the  front  and  create  a  stronger  reserve. 

R:  So  at  the  general  level,  all  your  antennae  are  out  to  see  if  there's  any 
counter- indications  for  your  primary  hypothesis  (2). 

E:  The  system  is  slow  enough  in  operating  that  I  cannot  say,  "I  want  you  to 
fly  out  and  do  this  and  give  me  an  answer  in  the  next  ten  minutes.”  So  you 
have  to  live  within  the  time  scale  of  which  things  can  be  done.  But  at  the 
same  time  I  would  probably  energize  the  system  by  some  direct  calls  to  people 
who  are  doing  the  collection  and  say  "Look,  be  particularly  aware  of  what  the 
ninth  tank  division  is  doing." 

J:  That's  these  guys  here  [pointing  at  map]. 

E:  Yes. 

J:  This  is  an  overview  of  what  the  state  of  defense  is,  it's  not  the  way  it's 
aligned  now,  roughly  how  the  enemy  is  in  line. 

E:  As  well  as  the  course  [?]  of  the  forces  that  are  in  contact. 

J:  Right.  These  units  are  bigger  than  these  units  though.  I  mean,  even 
though  they're  depicted  as  the  same,  so  bear  that  in  mind  as  well.  This  is 
the  ninth  that  General  Whitehead’s  talking  about. 

E:  I  would  also,  of  course,  give  him  what  the  force  ratios  now  along  the 

front  between  red  and  blue.  I  would  have  given  him  that  (3). 

R:  That's  in  the  intelligence  summary. 
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E:  I  would  be  trying  to  sensitize  s  little  taore  the  collection  and 
surveillance  plan  that's  already  in  notion.  In  other  words  you,  there  are 
only  a  few  things  that  you  could  change  at  this  late  date  and  generally 
they're  aircraft-related,  that  could  help  you  any  more  than  you're  now  being 
helped  by  the  collection  plan  you  have.  But  you  would  sensitize  the  people 
who  analyze  data  to  give  you  the  indicators,  as  quickly  as  they  can,  of  any 
change;  and  you  would  sensitize  some  major  subordinates  that  you  don't  have 
any  control  over,  the  brigades,  division  artillery,  etc.  So  that  this  is 
going  to  be  beyond  a  watch-out  [?],  because  this  is  going  to  be  a  sensitive 
period  while  you're  withdrawing  some  forces. 

R:  In  tracing  the  next  step,  could  you  let  us  know  what  you  think  of 
importance  might  happen,  and  then  pick  one  of  them  and  assume  that  it  does 
happen . 

What's  the  next  surprise  or  interesting  piece  of  information  that  might  affect 
what  you,  the  G-2,  do? 

E:  There  are  several  courses  of  action  the  enemy  can  have.  One  is  that  he 
could  continue  his  attack  with  the  forces  he  now  has  in  contact  or  with  a 
portion  of  the  forces  he  now  has  in  contact. 

R:  And  you'd  learn  about  it? 

E:  And  we  would  feel  that  all  the  time.  The  front  line  units  would  report 
that  they're  under  attack,  they  would  indicate  how  many  losses  they're 
receiving,  they  would  indicate  how  much  the  enemy  [is  moving?]  there,  whether 
they're  stopping  ... 

R:  So  you've  got  a  changing  sense  of  what  they're  up  to. 

E:  Yes.  Of  course,  another  course  that  could  be  happening  is  the  ninth  tank 
division  could  in  fact  be  committed  in  the  attack  through  one  of  those 
divisions . 

R:  And  again  you'd  learn  of  it. 

E:  You  would  either  learn  of  it,  as  it  was  moving  forward,  or  you  would  learn 
of  it  when,  in  the  worst  case,  they  came  into  contact  and  you  captured 
somebody  from  the  ninth  tank  division  in  late  afternoon  and  found  out  they'd 
been  in  combat  all  day.  You  get  fired  at  that  point. 

R:  If  that's  reasonably  likely,  (and  I'd  like  you  to  limit  yourself  to  a  pass 
through  possible  histories  that  are  reasonably  likely)  do  you  want  to  assume 
that  it  happened?  Or  is  that  really  an  outlier  of  something  that's  not  on  the 
most  likely  track.  What's  the  next  thing  you  think  you  might  learn? 

E:  Probably,  if  a  division  was  moving  forward,  you  would  get  some  indication. 
Remember  it  has  a  thousand  vehicles  in  it  or  something  of  that  order,  750  to 
1000  vehicles.  So  you  ought  to  learn.  It  has  lots  of  radios,  it  has  all 
sorts  of  other  signatures.  You  should  get  some  Indication  that  it's  moving. 

R:  Do  you  want  to  assume  that  you  do  get  that  indication? 
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E:  And  you  probably  would  by  naybe  10:30  In  Che  morning,  if  our  scenario  is 
that  that's  what  happened:  they  are  attacking  ... 

R:  So  within  half  an  hour  . . . 

E:  Let's  say  they  had  attacked  actually  at  8:00,  or  rather  at  10:00.  By 
10:30  or  11  you  would  have  some  Indication  of  a  heavier  attack  somewhere.  You 
would  also  know  they  were  moving  forward.  You  would  then  probably  arrive  at 
the  judgment  that  the  ninth  tank  division  is  probably  attacking  us  in  a 
specific  area. 

R:  OK.  Do  you  want  to  pick  an  area? 

E:  Make  it  in  the  center,  sort  of  on  the  boundary  of  the  two  brigades,  and.. 

R:  Has  it  passed  through  the  first  echelon  of  the  enemy? 

E:  It's  passing  through  right  now.  That  could  take  1  to  20  hours.  If  it 
succeeds  in  making  a  good  penetration  quickly,  it  would  pass  through  very 
quickly.  If  in  fact  you  stop  them,  then  they  would  be  intermingled  with  some 
first  echelon.  So  that's  why  I  say  I'm  uncertain.  What  you'd  be  trying  to 
find  out  was  whether  or  not,  how  much  damage  you've  been  doing  to  them,  where 
their  main  effort  is. 

?:  I  have  a  question.  By  first  echelon  you  mean  first  echelon  of  the 
division,  right? 

E:  Those  regiments  or  divisions  which  were  in  contact  already  and  fighting. 

?:  You're  thinking  of  the  red  first  echelon. 

E:  And  the  first  echelon  here  is  those  two  divisions  which  have  been  fighting 
in  contact  all  of  last  night  and  the  day  before.  The  second  echelon  division 
is  that  ninth  tank  division,  which  we  now  find  is  attacking  . . . 

R:  What  are  they  doing  that  we  learn  about? 

E:  We  saw  them  moving  and  one  of  the  brigades  has  reported  heavy  attack. 

Heavy  artillery  fire,  air  strikes  in  support  of  it,  on  one  or  two  battalions 
as  opposed  to  all  the  way  across  the  front,  with  lesser  fires  on  the  other. 

The  key  there  for  them  is  to  pick  out  where  the  main  effort  is.  They  may  have 
things  falling  on  all  across  the  front,  but  where  they  want  to  find  out  is 
where  the  heaviest  attack  is.  And  they  would  be  beginning  to  sense  that. 

R:  Is  there  any  chance  worth  thinking  about  that  they  would  try  to  fool  us? 

To  produce  preparatory  fire  power  in  the  wrong  place  in  order  to  make  us  think 
the  attack's  going  to  come  through  there.  Is  that  a  possibility  worth 
thinking  about  in  your  scenario?  I'm  looking  at  the  case  where  what's 
happening  is  different  from  what  we  think. 

E:  That  is  very  likely  to  happen  in  a  set  piece  attack,  where  the  lines  have 
been  stabilized  for  some  time.  It  Is  less  likely  to  happen  here,  because 
they've  really  been  moving  forward  all  the  time.  So  they  haven't  stopped, 
gone  through  the  planning  sequence  of  a  special  plan  which  would  have  that 


deception;  in  it  though  they  could,  they  still  could.  So  that's  possible,  but 
I  would  say  here  less  likely.  Again  they've  got  a  supply  situation,  say  in 
artillery  amaunitions,  so  you  nay  want  to  give  what  limited  supplies  you  have 
priority  to  the  forces  who  are  conducting  the  attack. 

R:  But  that's  not  a  G-2  decision,  that's  a  commander's  decision,  isn't  it? 

E:  The  G-2's  decision  is  whether  or  not  artillery  fire  over  on  the  right 
flank  is  the  part  of  a  deception  or  not,  and  it  is  the  G-2's  responsibility  to 
try  and  interpret  what  that  means. 

R:  In  your  dream,  do  you  want  to  assume  that  that  does  or  doesn't  happen,  and 
only  say  "yes,"  if  it's  somewhat  plausible. 

E:  I  would  say  "yes,"  but  let  me  say  it  in  a  slightly  different  context. 
Everybody  will  be  given  the  order  to  attack,  the  breakthrough  force  will  have 
more  artillery,  more  weapons  ... 

R:  This  is  the  enemy? 

E:  The  enemy,  yes.  Everywhere  along  the  front  they  will  be  given  the  order 
to  carry  out  secondary  attacks.  The  primary  attack,  let's  say,  is  in  the 
center  here.  It  will  have  more  artillery  with  it,  it  will  have  more  combat 
vehicles  in  the  physical  attack,  and  so  the  secondary  attacks  can  confuse  you. 
One,  confuse  you  as  to  where  the  main  attack  is  for  a  while;  and  the  secondly, 
it's  used  to  hold  your  forces  in  place  so  you  don't  use  them  as  part  of  your 
reserve.  So  that  will  happen,  but  if  they  begin  to  make  a  penetration  of  your 
lines,  you  should  know  pretty  soon  where  the  main  attacks  are,  or  at  least 
where  the  successful  attacks  are. 

R:  So  what  are  you  positing  in  your  dream  does  happen? 

E:  That's  right  there  would  be  secondary  attacks,  yeah,  O.K. 

R:  There  are  secondary  attacks,  and  we  know  that  they're  secondary  attacks. 

E:  At  least  the  only  penetration  is  in  one  area  and  therefore  we  assume,  - 
from  the  sightings,  radio  contact,  rate  of  fire  and  everything,  and  the 
gravity  of  the  situation,  -  that  that's  the  main  attack. 

R:  And  we're  confident  that  we're  right. 

E:  You  would  want  to  pin  down,  again,  the  order  of  battle,  how  many  regiments 
are  in  it,  or  how  may  divisions  are  in  it?  You  would  like  to  make  sure  that 
you're  not  being  fooled  and  it's  only  a  single  regiment,  and  actually  the  rest 
of  the  division  is  about  to  attack  from  a  different  angle  6  hours  later.  In 
other  words,  premature  diminishing  of  your  reserves. 

R:  What  would  you  do,  if  anything,  to  address  that  possibility? 

E:  As  part  of  the  standard  operating  procedure,  you'd  be  trying  to  find  out 
who's  in  that  attack,  the  order  of  battle,  and  ... 

R:  And  you'd  do  that  with  your  CEWI? 
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E:  Well  you'd  hope  very  hard  that  the  defense  would  take  prisoners,  and  you'd 
get  their  unit  identification.  You'd  hope  to  hear  on  the  radio  nets,  at  least 
the  nuaber  of  nets  that  are  up.  You  nay  not  hear  anything  that  you  can  get 
infomation  from.  You  would  hope  that  if  you  picked  out  certain  kinds  of 
artillery  battalions  that,  by  George,  that's  aore  of  those  battalions  here 
than  were  here  before,  so  that  must  be  a  new  division,  aain  attack.  You'd  be 
looking  for  all  the  snail  signals  and  pieces  of  information  that  would  help 
you  make  a  conclusion  "By  George,  the  whole  Red  division  is  in  that  attack." 

An  then  the  deeper  the  penetration  would  get,  the  more  confidence  you  would 
have  that  it  is  a  major  attack  (4). 

R:  In  the  process  of  keeping  your  eyes  open  for  these  developments,  would  you 
take  any  non-obvious  initiative,  for  example,  in  marshalling  your  collection 
resources,  or  would  they  just  do  their  normal  thing  and  the  word  will  come 
back  without  your  giving  any  special,  non-obvious  instructions? 

E:  Some  of  your  collection  resources  are  with  the  brigades,  and  the  question 
is  whether  you  would  take  them  away  from  them  or  not.  If,  in  the  initial 
penetration,  some  of  those  were  destroyed,  you  would  make  some  adjustments 
just  to  make  up  for  lost  forces  so  that  you  kept  majority  of  resources  on  the 
priority  problem. 

R:  But  this  would  be  a  maintaining  decision  rather  than  a  shift  . . . 

E:  Then  remember  now,  I  have  concentrated  on  the  ninth  tank  division  for  some 
time,  so  what  I'm  really  asking  them  to  do  is  a  continuation  of  the 
concentration  of  the  effort.  I  guessed  right  in  this  case,  so  I'm  really 
continuing  the  effort  to  know  more  about  the  location  and  activities  of  the 
ninth  tank  division,  so  I  would  tune  it  a  little  bit. 

R:  What  I'm  looking  for  is  places  where  a  poor  G-2  might  make  a  mistake.  So 
we're  identifying  places  where  your  decision  is  a  non- trivial  one,  and 
therefore  there's  something  to  be  learnt. 

E:  I  think  rather  than  saying  he  would  make  a  poor  decision,  I  think  it  would 
be  his  ability  to  use  all  the  -esources  available  to  him  and  adjust  them  in 
the  optimal  way,  some  of  which  he  controls,  some  of  which  he  doesn't,  and  to 
pull  the  information  out  of  the  system  and  to  adapt  the  system  to  do  a  better 
job,  as  opposed  to  being  poor  or  good. 

R:  Can  you  give  an  example  here  of  what  you  might  be  faced  with,  and  what 
decision  you  would  make  in  the  light  of  it,  so  we  get  an  example. 

E:  You  might  ask  for  some  of  the  aviation  brigades  scout  helicopter  resources 
to  be  made  available  for  you,  to  help  you  determine  the  size  of  the 
penetration.  You  might  talk  to  the  division  artillery  commander,  to  make  sure 
he's  concentrating  his  field  artillery  and  airborne  observers  on  that. 
Remember,  he  will  also  be  doing  that  with  his  forces.  You  might  ask  that 
addition  surveillance  resources  be  made  available  from  corps  from  their  . . . 

R:  I'm  assuming  that  all  these  mights  do  occurs;  in  other  words,  they're  in 
your  dream. 


E:  Yes,  they're  in  ay  dreaa.  Let's  say,  I'a  now  getting  into  the  G-3's 
business,  but  I  would  get  to  the  G-3,  and  we  aay  want  to  establish  a 
surveillance  line,  or  a  reconnaissance  and  security  line,  somewhere  outside 
the  penetration,  but  where  we  think  the  penetration  is  going. 

R:  This  is  an  aircraft  surveillance  line,  or  a  radar  . ..? 

E:  It  would  probably  be  the  reconnaissance  squadron.  It  has  both  a  good 
observation  capability  from  the  ground  and  air,  and  he  also  has  a  fighting 
capability.  So  the  G-3  would  be  interested  in  establishing  some  sort  of  a 
delaying  position,  and  I  would  be  interested  in  their  ability  to  observe  and 
report  what's  going  on. 

R:  And  this  is  part  of  your  own  CEtfl  battalion? 

E:  This  is  not  part  of  the  the  CEWI  battalion.  In  other  words,  I  would 
control  it  if  they  maneuver  for  it,  which  belongs  to  the  G-3,  which  is  in 
reserve.  And  I  would  say  maybe  we  can  get  some  of  their  [resources?],  since 
they're  in  reserve,  lets  use  their  air  cavalry  squadron  forward  to  do  some 
reconnaissance  rather  than  having  them  sit  back  there  in  reserve. 

R:  Is  there  some  chance  that  the  G-3,  for  reasons  other  than  Intel,  might  not 
go  along  with  it? 

E:  Very  likely.  Suppose  in  the  last  three  days  they’ve  lost  half  their 
helicopters,  and  he  knows  he's  going  to  have  to  use  them  to  fight  with,  and  he 
knows  he  needs  a  scout  helicopter  with  every  attack  aircraft.  So  he  doesn't 
want  to  let  the  scouts  move  forward  and  lose  some  more  of  them,  so  for 
conservation  of  resources,  or  maintenance  problems,  or  flying  hours  (our 
pilots  have  been  flying  15  hours  a  day),  or  any  number  of  reasons,  he  may  not 
choose  to  do  that. 

R:  Do  you  want  to  assume  that's  happened? 

E:  Let's  assume  that's  happened. 

R:  Incidentally,  if  anybody  else  gets  some  ideas  or  questions  to  ask  about 
this  unfolding  scenario,  please  do. 

K:  Just  one  interjection;  that  we  might  want  to  consider  not  just  the 
situation  where  Ennis  get  it  right,  but  the  situation  where  he  gets  it  wrong 
at  a  later  time. 

R:  Time  out  procedurally.  The  approach  that  this  is  introducing  is  one  where 
eventually  one  would  try  a  number  of  different  paths  through  the  possible 
future  that  test  the  interesting  alternatives,  but  at  the  moment,  to  get 
fixed,  we're  taking  something  like  a  modal  pass  (5). 

E:  Let  me  describe  the  general  situation,  and  then  we'll  change  from  what  I 
thought  was  happening  to  another  case.  There  has  been  some  sort  of  an  attack, 
which  was  heavier  than  the  previous  one,  which  led  us  to  believe  the  ninth 
tank  division  was  in  fact  attacking.  The  division  has  begun  to  take  the 
actions,  as  if  that  were  occurring.  At  the  same  time,  the  G-2  is  trying  to 
understand  more  about  what  the  ninth  tank  division  is  doing.  He's  still 
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watching  the  seventh  tank  aray,  and  he  has  also  got  surveillance  in  other 
areas;  so  that,  let's  say  two  hours  later,  all  of  a  sudden,  on  the  north 
flank,  the  ninth  tank  division  shows  up.  The  previous  attack  was  a  secondary 
attack  started  early  by  the  reserves  and  the  divisions  in  contact,  and  it  was 
not  the  ninth  tank  division;  and  now  the  ninth  tank  division  is  attacking  on 
the  north  flank. 

Now  see,  some  of  the  resources  are  still  scattered  throughout  the  division 
area  for  collection  of  intelligence,  some  have  been  concentrated.  Say  the 
signal  intelligence  people  have  been  concentrating  on  DFlng  signals  in  this 
area.  You  would  now  shift  that.  You  made  a  mistake  in  concentrating  on  what 
turned  out  to  be  the  secondary  effort.  If  you  began  to  get  some  signs  of 
movement  ahead  of  time,  hopefully  before  they  attack,  and  it's  enough  movement 
that  you  say  "now  I'd  better  reassess  my  estimate”,  and  then  you  finally, 
grudgingly,  reach  the  conclusion  that  you  may  have  made  a  mistake  and  that 
there  is  some  sort  of  attack  now  happening,  or  about  to  happen,  on  the  north 
flank.  You  would  adjust  those  collection  resources  of  your  own  which  can 
easily  be  adjusted,  all  are  not  adjustable,  but  those  that  can  be  then  put  on 
that  would  be  diverted  to  that. 

R:  Could  you  give  a  little  more  concreteness  about  how  precisely  you  might 
have  reassigned  your  resources? 

E:  You've  got  a  company  of  surveillance  radars.  They  are  not  easily  moved 
around.  You  usually  have  one  platoon  with  each  brigade,  and  they'd  probably 
stay  where  they  are.  They're  too  hard  to  get  in  and  out.  You  have  three 
helicopters  which  can  collect  non- communications  signals,  i.e.  radar  signals, 
[?]  signals.  You'd  probably  put  those  up  there. 

John:  So  you're  saying  you're  moving  ground  based... 

Ennis:  O.K.  we've  moved  the  ground  based  centers  up  and  then  there's  two 
other  things  I  would  do.  I  would  again  ask  for  help  from  corps  to  concentrate 
some  of  its  movable  resources  to  look  at  that  problem. 

Rex:  Does  it  have  its  own  CEWI? 

Ennis:  Corps  has  a  CEWI  brigade.  It'll  have  one  or  more  ground  battalions 
and  one  airborne  battalion,  which  has  a  pretty  good  collection.  So  I  would 
ask  for  corps  because  the  corps  headquarters  is  as  interested  in  the  main 
attack  as  we  are.  So  he  will  be  following  it  Independently  and  we  would  have 
probably  been  conversing  and  he  may  have  even  have  had  a  tip  off  that  it  was 
coming  and  I  missed  it. 

Rex:  Like  from  satellite? 

Ennis:  From  something  like  that,  some  other  way.  I  would  also  have  been 
staying  in  good  contact  with  the  division  on  my  left,  the  German  division,  and 
I  would  certainly  want  to  talk  to  them  right  now,  to  see  what  they're  holding. 
To  see  if  they  have,  through  their  sources,  found  out  that  yes,  in  fact  the 
ninth  tank  division  is  attacking  right  on  the  northern  boundary  of  our 
division  and  on  their  boundary.  So  you  need  to  also  talk  to  other  people  who 
have  collection  resources  and  who  make  individual  independent  estimates,  but 
they  tend  to  be  overlapping.  The  corps  is  looking  at  everything  I  am  and 


A-40 


nore,  the  German  division  follows  my  activities  very  carefully,  because  it's 
going  to  affect  him,  and  I  follow  his. 

R:  Can  we  tie  this  down  to  time?  Vhere  about  are  we  in  the  unfolding 
scenario? 

E:  Mid-afternoon.  Mid  or  late  afternoon.  In  other  words  at  10:00  or 
ten- thirty  there  was  some  sort  of  attack  in  the  middle.  Ue  thought  it  was  the 
ninth  tank  division.  The  division  commander  has  not  used  his  reserve,  but 
he's  reviewing  his  counter-attack  plan,  blocking  plan.  I  have  begun,  as  the 
G-2,  to  move  resources  to  better  understand  what's  in  there.  Three  -thirty  in 
the  afternoon,  I'm  clearly  getting  some  other  indications  up  north  which  I 
didn't  pay  as  much  attention  to  in  the  beginning  as  maybe  I  should  have. 

R:  From  outside  the  division? 

E:  They're  coming  up  on  the  northern  flank  of  the  division,  and  now  there's  a 
sudden  heavy  attack  there  and,  by  George,  they  say  they  captured  a  prisoner 
from  the  ninth  tank  division. 

K:  Can  we  back  up  a  minute?  Was  there  anything  you  would  have  done  at  ten  AM 
had  you  [envisioned?]  this  possibility? 

E:  If  I  thought  the  northern  flank  was  more  serious,  I  might  have  even  made  a 
trip  up  to  the  panzer  division,  to  talk  directly  with  their  G-2,  because  it's 
right  on  our  boundary.  Or  I  would  send  somebody  I  had  great  confidence  In  to 
talk  eyeball  to  eyeball  with  them.  There  is  a  difficulty  of  getting  written 
reports  over  the  teletypewriter.  Voice  direct  contact  is  very  good.  So  I 
might  have  had  closer  contact,  more  frequent  exchange  of  information  with  the 
panzer  division,  because  of  the  sensitivity  of  the  planning.  I  might  have, 
knowing  that  you  can  wear  out  your  flying  resources,  you  can  fly  just  a 
certain  number  of  hours  per  day  in  theory,  I  might  not  have  put  up  some  of  the 
helicopters  up  that  early  (6). 

R:  With  human  fatigue? 

E:  Both  aircraft  and  pilot,  yeah. 

Marvin:  Is  there  not  a  secure  phone  that  you  can  talk  to  G-2  on  in  that 
situation? 

£:  Yeah,  if  I  thought  It  was  serious,  I'd  go  to  the  trouble  of  flying  someone 
up  there,  I  would  certainly,  by  secure  phone,  do  it.  The  G-2  might  not  speak 
English,  I  may  not  speak  German,  I'd  probably  find  somebody  I  had  confidence 
in  who  spoke  good  German  go  up  to  their  map  [?],  talk  to  them,  get  a  sensing 
of  the  feel;  get  both  an  impression  and  factual  information.  If  I  had 
realized  earlier  that  that  was  the  critical  area,  or  if  I  had  estimated  that 
that  was  the  critical  area  (7). 

6 .  PROCEDURAL  INSIGHTS 

Claire  Cardie(SST):  All  the  readjustments  and  resourcing  that  you  do,  is  that 
something  that  you  learn  in  training? 
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E:  No,  you  learn  vhat  their  capabilities  are  in  school,  you  learn  what  their 
limitations  are  in  school,  and  through  your  experience,  you  then  are  trying  to 
apply  them  in  a  reasonable  fashion.  Much  of  it  is  cut  and  dry.  There  are 
sort  of  rules  of  thumb,  but  clearly  those  intelligence  collection  resources 
which  have  the  greatest  mobility  which  can  be  moved  around  and  all,  those 
would  be  the  ones  you  tend  to  play  with.  Those  which  do  not  have  a  great 
mobility  or  can't  see  deep,  they  would  tend  to  be  with  the  first  brigade, 
second  brigade,  third  brigade  and  just  stay  with  them. 

R:  In  what  we've  covered  so  far  have,  there  been  any  spots,  any  places  where 
a  G-2  might  make  a  mistake  and  therefore  where  it's  useful  to  know  what  an 
expert  would  do?  Or  are  most  of  the  decisions  you  are  making  ones  that  you'd 
have  no  concern  that  the  local  G-2  would  make  right,  and  therefore  wouldn't 
need  your  input  (1)? 

E:  I  thing  experience  tells  us  is  that  G-2s,  because  they're  working  on 
limited  information,  are  not  able  to  give  perfect  estimates,  ever.  And  then 
they  go  downhill  from  there  to  where  you  can  make  very  much  incorrect 
estimates.  The  Battle  of  the  Bulge  was  an  example  of  where  they  refused  to 
interpret  what  they  saw  in  the  way  in  which  the  Germans  actually  employed 
their  forces. 

R:  Would  we  then  have  to  have  defined  you  scenario  in  more  detail  in  order  to 
be  able  to  say  "aha,  this  what  an  experienced  guy  could  do  that's  different 
from  what  a  less  experienced  guy  would"?  Would  you  have  to  know  more 
precisely  what  information  he  got,  and  what  inferences  he  made  from  it  rather 
than  this  somewhat  broader  description  that  we're  handling  now  (2)? 

E:  You  could  give  me  a  detailed  scenario  and  I  could  make  bad  estimates  or 
the  wrong  estimates,  or  incorrect  or  slightly  off  the  center.  The  shortcoming 
of  that  vs.  doing  it  in  reality  is  the  paper  vs.  the  noise  and  all  the  other 
information  that  comes  in.  In  other  words,  when  you  put  it  on  paper  you 
narrow  it  to  the  essentials.  In  addition  to  that  there's  just  a  lot  of  other 
things  going  on  which  could  cause  you  background,  which  would  also  have  an 
influence  and  you  can't  represent  that  in  a  paper  exercise. 

R:  I  understand. 

E:  It's  less  than  totally  satisfactory,  understand. 

R:  Let  me  rephrase  it  a  little  bit.  This  is  a  procedural  time  out.  For  this 
exercise  to  be  useful,  it  seems  to  me  that  there  have  to  be  at  least  some 
places  where  the  observer  of  your  scenario  gets  some  insight,  says,  "Aha! 
that's  interesting.  I  wouldn't  have  thought  to  do  that.  And  I'm  wondering 
whether  the  level  of  our  questions  and  you  answers  have  the  right 
granularity;  whether,  in  order  to  get  these  kinds  of  insights,  you'd  have  to 
go  much  more  micro  than  perhaps  it's  feasible  for  us  to. 

Alternatively,  do  you  think  at  this  level,  we  or  a  new  G-2  could  be  learning 
interesting  lessons,  or  perhaps  will  be,  as  we  go  a  little  further  into  the 
scenario?  The  negative  would  be  somebody  reading  it  and  saying,  "so  what?". 
That  would  suggest,  either  that  this  isn't  a  good  approach,  or  that  we  need  a 
finer  grain,  or  that  we  need  to  ask  much  more  pointed  questions  along  the  way, 
than  the  rather  general  ones  that  we  are. 
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CC:  Is  it  a  problem  of  us  not  knowing  enough  to  be  able  to  say  "aha,  I 
wouldn't  have  thought  of  that"?  I  mean,  we  don't  know,  I  don't  know,  what's 
coming  across  here  that  might  be  different  than  what  I  would  have  learned  in 
training,  or  slightly  different.  I  know  that  I've  just  been  hearing  this  all 
and  accepting  it  all.  I'm  not  surprised  because  I  have  no  anticipations  of 
anything . 

J:  I  can  give  an  indirect  assessment  of  that.  Earlier  General  Whitehead  was 
going  over  the  Fulda  Gap  scenario,  or  the  Treysa  Defense,  as  Rex  has  renamed 
it.  I  was  comparing  the  kinds  of  things  that  he  was  saying  with  the  kinds  of 
things  that  the  lieutenant  colonels  were  saying.  Now  obviously  I  can't  read 
their  mind  to  say  well  aha  that  they  would  have  said  that,  but  comparing  the 
two,  some  of  the  considerations  that  General  Whitehead  had  were  very 
different.  In  fact,  he  pointed  out  things  that  none  of  the  lieutenant 
colonels  had  pointed  out.  And  they  were  not  at  the  micro  level,  but  in  terms 
of  general  considerations.  So  again,  I  don't  know  if  that  was  in  the  back  of 
the  lieutenants  colonel  minds  or  not,  but  it  didn't  come  out  in  what  they  were 
saying. 

E:  I  better  comment  on  that,  though.  The  lieutenant  colonel  was  the  G-3. 

R:  What  was  the  context? 

E:  The  context  was  the  choices  of  courses  of  action.  The  lieutenant  colonel 
was  being  the  G-3.  He  took  from  me  what  I  had  given  him  as  the  gospel.  I,  as 
the  new  G-2,  came  In  and  challenged  the  gospel.  So  remember,  I  was  playing  a 
different  role.  I  did  play  a  little  bit  of  the  G-3,  but  generally  what  a  lot 
of  my  ideas  were  raised  on  were,  "I'm  not  satisfied  that  I  had  it  right  as  the 
G-2."  Therefore,  I  was  going  "what's  my  biggest  threat  there?  What's  my  next 
biggest  threat  there?"  and  so  forth.  So  I  took  off  on  the  G-2's  track  as 
opposed  to  the  guy  who's  selecting  the  course  of  action. 

R:  I  suppose  what  my  question  boils  down  to  is:  are  we  uncovering  any 
difficult  choices,  and  are  we  getting  into  the  right  kind  of  detail  that  would 
help  us  understand  why  you  advocated  the  choice  that  you  did.  I  have  a 
concern  that  we  may  not  be.  Or  perhaps  we're  too  early  on  in  the  script  and 
further  downstream  as  we  allow  you  to  play  out  your  dream  there'll  be  more 
telling  incidents.  It's  the  telling  incidents  and  how  you  handle  them  that  I 
think  we'll  find  is  helpful. 

E:  I'm  worried  that  we're  not.  It's  sort  of  like  the  father  and  son  thing. 
The  son's  going  to  have  to  judge  whether  he  hears  and  what  the  father  makes 
sense.  Everything  the  father  says  is  not  right.  The  world  will  be  different 
from  the  father's  world,  so  there  are  good  kernels  in  there  and  there  are 
probably  bad  kernels  of  knowledge.  With  experience  being  overlaid  on  top  of 
book  learning  and  not  all  experience  is  right. 

R:  For  this  to  be  a  successful  knowledge  elicitation  technique,  reading  or 
hearing  the  sequence  needs  to  be  in  some  sense  a  surrogate  for  being  there. 
This  obviously  Isn't. 

E:  The  methodology  here  is  good;  I'm  worried  ab  it  the  general  conclusions. 

In  other  words,  once  you  get  above  the  factual  level  and  get  into  the 
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judgmental  levels  you  change.  If  somebody  else  interprets  a  condition 
differently  then  his  response  could  be  as  good  as  or  better  than,  or  worse 
than  [yours]. 

R:  That  I  think  doesn't  matter,  even  though  what  you  would  do  isn't 
necessarily  gospel.  If  we  understand  why  a  seasoned  G-2  made  an  important 
decision,  that  [a  trainee]  might  have  made  differently,  so  that  the  experience 
counts  for  something,  then  I  think  we'll  be  happy  as  far  as  our  purpose  is 
concerned.  Unless  anything  in  particular  occurs  to  you  there,  I'd  suggest  we 
proceed  a  little  further. 

K:  Before  we  leave  this  . . . ,  do  you  think  it  might  help  to  ask  Ennis  to  say 
is  there  anything  that  you  would  do  now  differently  then  you  would  have  back 
when  you  were  a  greenhorn  G-2? 

R:  Yes,  that  would  be  helpful.  Let's  address  that,  and  then  I  suggest  that 
we  pause  for  a  moment  to  let  you  finish  your  lunch  and  then  the  rest  of  us  can 
ventilate  whatever' s  on  our  minds.  Do  you  want  to  respond? 

E:  I  guess  in  answering  the  question  if  I  had  to  do  it  over  again,  what  I 

would  do  differently,  I  would  have  learned  my  trade  better.  I  would  have 
learned  the  facts  better  and  I  would  have  tried  to  also  anticipate  the  breadth 
of  the  job,  such  as  the  new  G-2  trying  to  understand,  above  the  factual  level, 
what  the  breadth  of  the  job  is.  And  so  I  would  say,  you  never  do  that  well 
enough.  I  must  say  I  probably  did  it  better  than  average,  but  still  not  good 
enough.  So  that's  a  dumb  answer,  but  it's  a  truthful  one.  You  suddenly  find 
out  you  didn't  work  hard  enough  on  some  of  these  things. 

K:  Is  there  any  specific  decision  that  you  might  have  made  differently  back 

then  than  you  would  now  and  why? 

R:  In  this  context? 

K:  Yeah. 

E:  I  would  have  studied  harder  in  Ft.  Huachuka  and  Ft.  Leavenworth. 

K:  No,  but  let's  say  that  when  you  first  came  on  line.  Is  there  something 
that  you  might  have  missed  seeing,  you  might  have  allocated  you  resources 
differently  and  been  sorry  about  it  later,  that  you  wouldn't  have  missed  this 
time  around? 

R:  What  good  has  your  experience  done  you? 

E:  Some  of  the  resources  would  be  allocated  cut  and  dry,  no  problem.  It 
would  be  to  make  sure  I've  used  everything,  full  up,  110%,  getting  the  best, 
with  slightly  different  allocation  and  new  thoughts  on  how  to  do  it,  but  then 
getting  out  of  the  system  110%.  In  other  words,  drive  the  system  harder, 
which  is  not  knowledge  (3). 

CC:  Harder  than  you're  trained  to? 

E:  Harder  than  they  ever  expected  to  do.  And  I  don't  know  what  that  means. 
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CC:  You  mean  harder  than  you're  taught  to  use  then  in  school? 

E:  O.K.,  let's  say,  the  easy  one  of  this  is  the  Flying  Hour  program,  in 
peacetime  there's  a  Flying  Hour  program,  with  so  many  hours  of  a  pilot  and  so 
many  hours  on  an  aircraft.  Of  course  that's  simply,  everybody  realizes,  the 
first  thing  you  do  is  start  flying  them  into  the  ground.  So  you  do  that,  but 
then  also  you  want  to  do  that  intelligently  so  that  you  don't  fly  them  into 
the  ground  and  you've  got  nothing  [left?].  So  you  then  come  back  knowing  your 
job  better  so  you  prioritize  and  don't  use  the  aircraft.  I'm  picking 
something  easy,  don't  use  the  aircraft  purposely  on  less  than  the  key 
missions,  if  you  know  that's  a  very  scarce  resource.  Don't  get  it  shot  down 
and  don't  get  it...  That's  all  tuning  it,  I  don't  know. 

R:  Let's  let  Ennis  finish  his  lunch  and  let's  entertain  any  discussions  that 
might  suggest  to  us  what  we  should  do  next,  including  a  mid- course  correction. 

Craig  Cook:  This  seems  to  me  to  be  almost  a  classic  case  study  approach, 

where  you  present  some  examples  and  somebody  just  kind  of  talks  about  this 

case  study.  The  only  thing  that's  different  here  is  it  didn't  really  happen, 
so  there  isn't  any  result,  O.K.,  which  makes  it  even  shallower  than  a  case 
study  approach. 

I  also  sense  that  the  vague  hypothetical  nature  of  the  whole  thing  doesn't 
allow  him  to  get  down  to  specific  details  of  what  it  is  that  he  make  a 
decision  about.  What  we're  hearing  is  general  strategic  level  kinds  of  stuff, 
which  having  said  it  once,  seems  like  it  should  be  enough.  But  I  suspect  the 

thing  that  makes  it  really  expertise  has  to  do  with  the  ability  to  use  it  at 

the  time  that  you  need  it  and  to  know  which  part  to  use  and  which  part  not  to 
use,  which  he  can't  verbalize,  probably,  but  can  only  do  through  some 
exercise.  So  1  think  it's  going  to  be  very  difficult  for  us  to  get  down  to 
deeper  expertise,  through  this  kind  of  scenario,  story  telling  without  putting 
him  in  a  real  working  [situation?]  whatever  you'd  actually  do  to  go  through 
that  kind  of  a  scenario  (4) . 

R:  Now  would  a  solution  to  this,  or  a  step  toward  a  solution  be  simply  to 
elaborate  the  information  so  that  Ennis  has  had  to  master  a  whole  book  full  of 
stuff  in  order  to  replicate,  without  being  there,  as  closely  as  you  can,  what 
he  would  really  know  on  the  ground.  I  suppose  we  could  ask  him  two  questions. 
One  is,  if  we  could  do  that,  if  we  could  actually  fill  in  all  the  supporting 
details  for  him  and,  as  he  moved  forward  through  the  scenario,  if  we  could 
again  paint  in  [the  details];  if  we  could  do  it,  would  that  have  answered  your 
question?  If  the  answer  is  no,  then  we  don't  need  to  worry  whether  we  can  do 
it. 

E:  May  I  comment?  Remember  when  at  Leavenworth  for  four  days,  we're  going  to 
get  a  chance  to  watch  students  do  the  same  thing  in  the  greater  detail  level. 
The  question  is,  whether  that's  going  to  be  worthwhile  or  not.  But  that  will 
respond  in  the  way  that...  Only  you  won't  be  able  to  talk  to  the  students. 

Craig:  Yeah,  we  can't  talk  to  them.  So  here  we're  asking  questions  and 
you're  providing  insights.  There  we're  just  observing  them  and  that's  yet 
another  technique  and  they  don't... 


A-45 


Kathy:  How  about  when  we  come  back  fron  Leavenworth,  we  sort  of  know  what  a 
map  exercise  looks  like.  Then  one  of  our  experimental  techniques  could  be  to 
give  a  map  exercise  with  interruptions  (5) . 

Theresa:  I  really  disagree  with  your  [CC?]  saying  this  is  all  being  done  on  a 
strategic  level.  There's  some  quite  detailed  [material].  Without  getting 
into  the  actual  nitty-gritty  of  the  pieces  of  equipment  that  you  look  at,  and 
sort  of  play  acting,  remember  we're  just  using  the  scoop  full  of  dirt  sort  of 
thing  here,  rather  than  trying  to  do  an  actual  excavation.  There  is  certainly 
a  process  that's  kind  of  going  on  here,  where  Ennis  will  describe,  in  general 
terms,  his  next  piece  of  scenario;  and  then  go  into  detail  about  what  he'll 
do.  And  sometimes  step  back  and  give  us  a  sort  of  hierarchial  explanation  or 
a  sort  of  "from  the  manual”  or  from  the  outside,  to  help  us  organize  it,  the 
sort  of  "by  the  book"  sort  of  thing,  step  back  and  give  us  an  explanation  like 
that.  You  know,  you're  getting  at  some  different  things  here,  and  if  this  is 
just  an  exercise  in  trying  to  see  what  we  can  get  taking  this  approach  I  think 
we're  getting  something.  I  don't  know  if  the  purpose  today  is  to  get  really 
detailed  structure. 

R:  To  capture  the  texture... 

K:  The  issue  is  whether  the  spoonful  of  dirt,  shallow  that  we're  using  today, 
can  really  get  at  knowledge  that  is  deeper  than  what  the  G-2  would  learn  at 
school.  The  kind  of  stuff  that  Ennis  could  give  a  G-2  that  he  wouldn't  learn 
in  school. 

T:  You  know,  I  think  the  problem  we  had,  that  I  mentioned  before,  is  that 
it's  hard  to  know  for  us  maybe  to  appreciate  that,  not  having  gone  to  school. 
Sometimes  I  think  that  maybe  there  are  things  insights  here  the  we,  because 
he's  the  first  person  we've  done,  maybe  we  should  have  started  out  with  the 
person  that  we  assume  that  we  are.  We  should  have  started  out  with  that 
person,  and  tried  to  get  what  they  know  then  we  can  see  the  difference  (6) . 

Craig:  Just  as  he's  pointed  out  heuristics  that  say:  "never  believe  all  the 
intelligence  summaries  that  you  see,"  also  there  are  heuristics  about 
knowledge  elicitation  which  say:  "don't  convince  yourself  that  you  have 
learned  everything  there  is  to  know  about  the  subject  from  an  early  pass  over 
it."  So  I'm  just  kind  of  cautioning  that  we  may  not  even  have  begun  to 
uncover  the  depth  of  knowledge  that's  required  to  be  expert. 

K:  I  think  that's  obvious.  I  mean  If  it  were  this  easy,  they  wouldn't  be 

paying  us  money  to  do  it. 

T:  Another  thing  that  might  help,  though,  even  to  give  us  a  sense  of  what,  if 

we  go  ahead  and  do  this  for  real,  what  sort  of  order  we  want  to  do,  or  how 

many  people  we  should  do.  I  mean,  we  need  to  get  a  sort  of  pool  of  these 
things;  we  need  a  sampling  of  people  going  through  this  exercise  and  maybe 
with  different  levels  of  experience.  That’s  something  else  we  might  work  on. 

Craig  Will:  Part  of  what  Craig  is  pointing  out,  I  think,  is  the  tangibility 
of  the  information.  My  impression  is  that  of  what  he's  said  so  far,  very 

little  of  that  would  we  actually  want  to  code  into  a  machine  (7).  Although 

everything  he  said  was  stuff  we'd  want  to  know  if  we  were  doing  that  coding 
process.  I  think  that  one  of  the  problems  you  gee  into,  is  you  start  actually 
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coding  stuff  into  a  machine  and  building  a  system  and  asking  him  very  specific 
tangible  information  and  showing  him  a  prototype  system  and  all  that  sort  of 
stuff.  I  think  that  my  impression  (probably  this  might  have  led  to  Craig's 
comment)  is  that  when  you  get  into  the  specifics,  the  expert  might  be  likely 
to  backtrack  on  some  of  the  general  statements  that  he  made  before  and  start 
saying  things  that  are  inconsistent  with  what  he's  told  us  so  far.  It's  a 
murky  issue  about  whether  what  he  thinks,  when  he  thinks  about  it  in  very 
general  terms,  his  model  of  how  he  does  things  in  the  general,  really  matches 
up  with  the  specifics  within  a  particular  case.  He’s  shuffling  this 
information,  mixing  it  with  that  information,  and  getting  this  output,  and 
then  all  of  a  sudden  everything  becomes  a  special  case.  "Oh  well,  it's  just 
this  one  little  case  over  here  that  I  don't  follow  that  rule".  It  gets  very 
confusing  and  very  murky,  and  that's  I  think  part  of  why  he  raised  these  types 
of  issues. 

E:  Can  I  comment  on  where  we're  going?  We're  developing  an  elicitation 
methodology,  or  a  family  of  them  or  a  set.  We  haven't  been  told  what  the 
application  is.  We've  been  told  to  search  for  that  application.  To  pick  up 
on  what  you  just  said,  the  Army  right  now  is  putting  a  number  of  work  stations 
for  people  to  put  in  what  I  call  the  educational  and  training  knowledge,  so  he 
can  then  work  on  it.  So  my  question  is,  should  we  be  concentrating  on  how  to 
best  put  the  training  information  that  people  take  into  that  workstation?  Or 
are  we  then  at  the  more  judgmental  level  of  handling  inconsistencies  and  that 
sort  of  thing.  In  other  words,  what  I  call  the  procedures  of  workstations 
with  all  the  training  agents  to  help  you  find  things;  or  are  we  concentrating 
on  this  judgmental  individual  who  has  people  assisting  him  at  the  structure 
level  but  he's  trying  to  make  sense  out  of  the  whole. 

R:  As  far  as  today's  exercise  is  concerned,  we're  not  addressing  the  case 
where  the  purpose  of  the  knowledge  elicitation  is  to  program  as  far  as  expert 
system;  though  we  certainly  will  be  as  far  as  that's  concerned.  The  purpose 
we're  addressing  today  is  a  training  purpose.  And  not  just  any  old  training 
purpose,  but  the  purpose  of  training  somebody  who's  got  the  standard 
information,  that  you  get  out  of  doctrine  and  training  and  seeing  whether  we 
can  tack  on  the  additional,  the  bonus,  expertise  that  somebody  with  experience 
can  have. 

In  the  particular  step-through  exercise  that  we're  playing  with,  I  agree  with 
Craig,  I  don't  think  it's  quite  working.  The  object,  that  maybe  can  be 
achieved  some  other  way,  is  to  try  and  observe  your  production  rules,  if  you 
have  any.  Part,  I  think,  of  what  we're  missing  here,  which  may  suggest  a 
different  tack,  is  that  there's  not  enough  circumstantial  detail  with 
guaranteed  realism  being  made  available,  first  of  all  to  you  (Ennis),  and 
secondly  to  the  reader  of  the  script,  who  presumably  is  the  one  who  will  learn 
from  it.  We're  assuming  that  a  set  of  scripts  like  this  can  be  useful;  either 
because  the  reader,  or  maybe  somebody  playing/repeating  the  role,  will 
inductively  learn  similar  production  rules  directly,  or  possible  will  generate 
a  more  interpretive  structure  of  his  own. 

In  either  case  it  seems  the  elicitation  needs  to  have  a  feature  that  we're  not 
getting  here.  This  is  the  richness  of  the  real  information  and  the  real 
circumstances,  which  will  discriminate  between  an  experienced  G-2's  handling 
and  a  novice,  which  is  what  we  really  care  about.  I  have  a  feeling  that,  at 
the  level  we're  handling  it,  the  reader  would  have  no  way  of  saying  that  this 
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was  a  novice  or  an  expert,  and  that  therefore  it's  not  capturing  what’s 
distinctive  about  the  expert's  knowledge. 

K:  I  think  what  we  need  to  do  is  to  have  two  Ennis's,  one  of  them  to  really 
play  the  game.  One  of  them  is  going  to  say  "Send  out  the  Mohawks”  and  the 
other  one  is  going  to  come  back  and  say  "the  Mohawks  have  reported  evidence  of 
movement” . 

T:  We're  looking  at  Ennis  the  expert  though.  I'm  wondering  if  we  should  have 
Ennis  play  two  Ennis's;  Ennis  before  [and  after]  the  expertise.  You  know  when 
you  were  asking  him  what  he  would  have  done  differently,  I'd  be  interested  to 
see  you  go  through  it,  almost  as  you,  if  it's  possible  to  ... 

Craig  Cook:  Is  there  such  a  data  base?  Does  there  exist  a  scenario  carried 
out  in  its  fullest  extent  with  the  troops  moved  from  here  to  there  it  took 
them  3.2  hours  and  they  engaged  this  set  here  and  ...  (8) 

R:  And  this  is  what  they  saw  on  their  radar  screens... 

Craig  Cook:  Now  is  it  possible,  if  ve  got  a  hold  of  that  whole  thing  and 
presented  it  to  you  in  some  form,  a  massive  printout,  a  computer  simulation, 
or  whatever,  that  you  could  then  issue  command  decisions,  "I  want  to  send 
radar  collection  over  at  this" ,  and  that  the  data  would  be  there?  If  you  did 
that,  this  is  the  data  that  you  would  get  back.  O.K.,  but  you  don't  see  the 
whole  thing,  you  just  see  the  results  of  what  it  is  that  you  asked. 

R:  Even  if  it's  a  human  playing  God  on  what  comes  back. 

Craig  Cook:  Well,  except  a  human  has  to  be  consistent,  O.K.  You  don't  want 
to  get  into  games  playing  and  making  up  creative  things  which  will  throw  off 
his  practicality. 

Kathy:  More  to  the  point,  we  probably  want  to  have  somebody  at  the  level  of 
just  out  of  school  issue  the  decisions  and  we  want  Ennis  kibitzing  over  his 
shoulder. 

E:  Remember,  the  idea  was  to  start  out,  figure  out  what  you're  going  to  do, 
go  to  the  school  system,  which  is  what  I  think  will  be  central  to  this,  and 
then  finally  go  out  and  look  at  what's  done  in  the  division,  which  will  turn 
out  to  be  a  disappointment. 

Craig  Cook:  There's  a  key  point  in  your  statement  which  is;  is  the  focus  of 
the  project  to  distinguish  elicitation  for  expertise  from  elicitation  for 
novices,  which  deals  with  what  is  the  nature  of  expertise  and  how  does  it 
differ  from  the  novice's  knowledge. 

Kathy:  Is  it  right  for  us  to  be  focusing  on  the  issue  of  what  makes  an  expert 

an  expert?  If  it  is  maybe  what  we  should  use  our  school  solutions  for  is  to 
give  the  experts  something  to  kibitz  over. 

R:  My  working  assumption  for  just  today's  exercise  which  may  make  it  quite 
unrepresentitive  of  the  total  project,  is  a  working  hypothesis  that  an 
experienced  G-2  knows  something  that  we  would  like  to  be  able  to  elicit  and 
put  in  a  holding  bin,  so  that  later  it  can  be  transferred  to  a  novice  (if  that 
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transfer  can  be  achieved).  This  project  only  addresses  the  elicitation  part 
of  it  and  not  the  transmittal  part.  But  If  the  total  process  can  be  achieved, 
it  will  achieve  something  that  the  Army  wants  to  achieve  and  it's  a  legitimate 
function  for  this  project. 

Marvin:  I'm  not  sure  I'm  following...  You  seem  to  have  a  goal  though  of  being 
able  to  elicit  just  those  parts  of  his  expertise  that  are  different  from  the 
novice's.  But  his  expertise  may  be  so  integrated,  and  the  whole  thing  may  be 
structured  differently,  that  you  can't  just  elicit  parts  that  are  different... 

R:  I  understand. 

Craig  Cook:  I  think  it  may  also  be  that  the  parts,  the  real  sophisticated 
parts  are  situation  dependent  and  can  only  be  elicited  under  a  given 
situation. 

Marvin:  I  think  there's  a  really  interesting  empirical  question,  which  I 
wouldn't  ask  Ennis  and  expect  him  to  know  the  answer,  which  is  whether  you 
really  use  these  general  heuristics,  in  other  words,  use  all  the  resources, 
110%,  question  the  estimate  that's  handed  to  you.  In  other  words,  do  you  have 
a  fairly  high  level  knowledge,  which  then  very  powerfully  generates  specific 
actions?  Or  do  you  have  a  lot  of  low  level  rules,  in  other  words,  use  this 
type  aircraft,  use  the  artillery  airborne  reserves,  use  the  helicopters? 
There's  a  real  difference  in  terms  of  how  your  expertise  is  really 
represented.  If  you  have  the  more  general  approach,  it  may  be  more  robust. 

If  you  are  given  new  collection  resources  in  a  situation,  you  might  adapt  more 
easily.  On  the  other  hand,  it  might  take  you  longer  to  figure  out  what  to  do, 
than  if  you  had  these  low  level  rules.  It  may  be  that  you  have  both.  But  I 
don't  think  it's  obvious  that  his  expertise  will  be  at  the  level  of  low  level 
rules,  there  may  be  some  level,  we're  just  getting  to  right  now  that  is 
legitimate.  I  wouldn't  dismiss  that  (9). 

R:  Just  a  clarification  of  what  position  I  was  representing.  You've  forced 
me  to  think  about  it.  I'm  not  suggesting  that  expertise  is  an  add-on,  but 
that  the  totality  of  what  the  expert  knows  represents  some  improvement,  that 
you'd  like  to  achieve,  over  what  the  novice  knows.  Even  if  there's  very 
little  overlap,  there  are  theories  which  suggest  that  novices  think  in 
structurally  quite  different  ways  from  experts,  even  the  experts  the  novices 
become.  But  I'm  assuming  that  we  want  what  the  expert  knows  in  totality,  and 
the  benefits  from  that  are  somehow  to  be  made  available  ultimately  to  those 
that  don't  have  the  expertise.  The  best  medium  for  achieving  it  may  be  to  put 
it  all  in  a  book,  in  a  training  program,  a  computer  . . . 

Marvin:  I  wouldn't  be  as  pessimistic  about  this,  I  thought  it  was  going 
rather  well.  We're  not  giving  him  the  details,  and  in  fact  he's  having  to 
generate  them  himself  and  perhaps  not  doing  a  great  job  generating  details. 

But  what  we're  getting  as  a  result  are  some  kind  of  general  guidelines.  Like, 
"I  would  shift  the  resource  if  I  had  been  wrong  about  where  they  were 
attacking".  Now  later  when  we  provide  details,  we  can  check  for  consistency 
and  see  if  there  are  two  levels  of  representation.  If  they're  different, 
which  one  then,  on  reflection,  you'd  say,  "yeah  this  one  is  better,  or  maybe 
they're  both  working  in  some  way,  but  we're  getting  something  here". 

R:  Oh,  yes. 
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CC:  I  would  also  think  that  Ennis'  ability  to  pop  in  from  one  level  to 
another  is  something  that  an  expert,  the  integration  of  their  lover  level 
strategies  to  a  higher  level  picture,  I  think  is  something  I  associate  with  a 
more  experienced  person  being  capable  of  doing.  I  would  almost  guess  that  we 
won't  find  that  ability  to  step  back  from  what  they're  doing  and  tell  you 
about  it  in  a  less  experienced  person. 

Kathy:  The  other  interesting  issue,  and  we  talked  about  it  in  other  meetings 
and  it  really  hasn't  come  up  here  is  what  is  an  expert  on  World  War  3.  Are 
there  any? 

R:  There  are  only  novices. 

CC:  Then  we  wonder  if  you,  because  of  your  past  experience  have  a  greater 
appreciation  . . . 

R:  An  advanced  novice. 

E:  Or  a  worse  appreciation,  because  I'm  captured  by  my  experience. 

CC:  You  have  a  certain  expectation  for  the  way  your  knowledge  will  degrade, 
or  things  will  degrade. 

E:  That's  right. 

Craig  Cook:  So  you  don't  know. 

CC:  I  have  a  couple  of  questions  about,  I  guess,  I'm  surprised  at  the  fluency 
that  you're  able  to  talk  about  this  stuff. 

Ennis:  I  haven't  been  caught  out  yet.  I  lie  a  lot. 

Craig  Will(?):  And  I  had  two  questions  about  this.  You  never  once  said  "gee 
whiz  1  haven't  done  this  specific  thing  for  10  years,  how  would  I  know  that." 
Now  1  guess  I'm  curious,  and  the  other  thing  Is  that,  well,  it's  a  question 
about  doing  vs.  training.  How  long  has  it  been  since  you've  done  this  sort  of 
task  and  how,  when  you  were  presumably  pretty  high  up  in  the  Army,  how  removed 
does  one  get  from  this  sort  of  thing?  Do  you  get  involved  with  this  sort  of 
stuff  enough  so  that  your  skill  and  knowledge  keeps  getting  updated? 

E:  I  haven't  fiddled  around  the  G-2  part.  Certainly  the  operations  and  what 
things  mean  on  maps  and  time  and  space  factors  and  all  that.  I've  done  enough 
time  over  the  years  to  feel  fairly  comfortable.  When  you  get  me  down  to  the 
details,  depending  on  how  much  I've  done  that,  I  get  less  and  less  comfortable 
and  less  and  less  accurate. 

CC:  You  see,  that  may  be  one  explanation  for  why  we  didn't  talk  about 
details,  if  we  brought  some  person  in  who  really,  who  did  it  yesterday,  he 
would  probably  want  to  tell  us  all  sorts  of  stuff  about  details,  and  maybe  we 
wouldn't  get  the  sort  of  high  level  stuff  that  we  got  from  you  (10). 

E:  Remember  again  that  you've  asked  me  to  be  everybody  on  the  G-2  staff.  I 
kept  trying  to  be  the  G-2,  I've  got  an  assistant  who  is  a  head  of  collection 
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management  who  I  say  "give  priority  to  that" .  I  aay  even  be  smart  enough  to 
say  "let's  get  soae  helicopters  up  there”.  He  will  in  fact  have  a  plan.  In 
fact  he  will  have  another  guy  working  for  him,  who's  on  duty,  who  will  adjust 
the  current  plan  and  he'll  coae  back  and  say  "we  can't  do  that  cause  we're 
doing  this  or  that".  So  you're  asking  me  to  be  everybody  in  a  group,  by  the 
way. 

R:  In  fact  we  only  need  you  to  say  what  you  would  do  as  the  G-2. 

E:  I  can  go  down  and  be  the  detail  guy. 

R:  I  think  we  do  not  want  you  to  be  other  than  the  G-2.  In  fact  how  you're 
underlings  implement  your  orders  is  part  of  the  malignant  outside  world  (11). 

E:  You  see  I  came  up  through  that.  I  was  an  underling,  I  was  the  collection 
manager  in  the  G-2  staff  or  the  corps  staff  somewhere  in  the  path.  So  I've 
done  some  of  those  other  jobs. 

CC:  The  other  question  I  had  is  a  question  of  training.  There's  a  difference 
between  the  people  who  are  experts  and  just  do  their  tasks  and  people  who  are 
teachers.  And  lots  of  experts  maybe  will  forget  lots  of  high  level  rules  that 
they  learned  because  they  just  do  their  task;  and  they  only  have  to 
reconstitute  them  in  order  to  train  other  people.  Now  in  your  experience  in 
the  Army,  is  Army  a  situation  where  there  are  experts  that  do  their  thing  and 
there  are  other  people  who  are  trainers,  or  is  it  typically  the  case  that 
anyone  who's  a  expert  also  spends  a  lot  of  his  time  training  other  people. 

E:  In  theory,  some  of  the  people  down  at  Huachuca  and  whom  we  should  run  into 

at  Leavenworth  have  been  the  experts,  have  been  division  or  corps  G-3s  or 
assistants  to  G-2s,  and  they  are  now  back  teaching. 

CC;  I  have  a  suspicion  that  when  you  have  an  expert  who's  also  a  teacher  you 
may  get  two  layers  of  knowledge,  two  layers  of  representation.  There's  the 
way  the  guy  really  does  it,  and  then  there's  the  way  that  the  guy  tells  other 
people  he  does  it,  and  the  second  layer  may  be  to  some  extent  phony  (12). 

E:  When  the  instructor  starts  telling  the  war  story,  that's  right,  and  gets 
away  from  the  text. 

CC:  Well,  or  it's  hard  to  articulate  something  or  you  need  to  oversimplify 
it  so  somebody  can  understand  it.  So  you  make  up  things  that  aren't  really 
literally  true  about  how  you  do  stuff,  and  you  figure  that  when  the  student 
gets  to  the  highest  level  of  expertise,  he'll  find  his  own  way  of  solving  that 
problem. 

7 .  KNOWLEDGE  STRUCTURE 

R;  I  think  I'm  going  to  want  to  move  us  along  before  we've  exhausted  these 
issues,  just  so  that  we've  covered  a  couple  of  bases.  Does  anybody  here  want 
to  try  out  any  ideas  from  your  particular  perspective,  Craig  or  Marvin  or 
John,  that  would  enable  us  to  use  this  as  a  test  for  either  knowledge 
structures  or  elicitation  techniques.  If  there's  not,  then  there's  something 
else  I  want  to  go  on  to,  but  if  there  is  anybody  who  would  like  to  test 
anything  out  I  would  like  to  give  priority  to  it. 
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John:  Yeah  I  would.  If  anybody  elae  wants  to  I'll  divide  ay  time  up. 

Anybody  else?  O.K. 

Marvin:  Are  we  meeting  again,  Rex? 

R:  That's  one  of  the  things  to  be  discussed.  I  think  you  and  John  need  to 
decide  whether  you  think  it  would  be  useful  or  not.  I've  committed  a  certain 
amount  of  resources  and  I'm  game  to  follow  up,  but  I  think  it's  your  judgments 
on  whether  [there  should  be]  a  next  step  along  these  lines.  I'd  be  interested 
in  doing  it,  but  I  think  that's  your  choice. 

John:  The  purpose  of  what  I'm  going  to  be  asking  you  now  is  to  try  to  get  a 
sense  of  overall  structure  to  what  some  of  the  things  you've  been  talking 
about.  You  talked  about  things  like:  wanting  to  know  where  the  enemy  threat 
is,  and  what  he  can  do  to  me,  and  where  he  is  and  so  on  and  so  forth.  I'm 
going  to  ask  you  a  very  general  question.  Why  does  all  this  matter? 

E:  It  goes  back  to  the  division  or  corps  mission.  I'm  on  the  division 
commander's  staff,  he  has  been  given  a  mission  to  defend  the  piece  of  ground. 
I'm  concentrating  on  what  I  thought  were  the  most  important  threats  to  doing 
his  mission.  That's  the  only  reason  it's  important,  mission  accomplishment. 

John:  So  mission  accomplishment  is  your  overall  goal  then  is  what  you're 
saying.  (1) 

Ennis :  Yes . 

J:  Then,  you're  looking  at  threats  to  this  mission  accomplishment. 

E:  There's  a  second  one  and  that  is  the  ability  to  carry  out  the  next 
mission.  In  other  words,  I  do  not  want  to  exhaust  all  my  resources  so  there 
is  no  capability  tomorrow. 

J:  O.K. ,  so  give  me  a  summary  statement  then.  When  you're  thinking  in  terms 
of  mission  accomplishment  and  next  mission.  What  is  it  that  you're  trying  to 
do  then?  I  mean,  give  me  a  sentence  that  describes  all  your  concerns. 

E:  I  am  trying  to  describe  for  the  division  commander  what  the  greatest 
enemy,  terrain,  weather  impacts  there  can  be  on  his  accomplishment  of  his 
mission. 

J:  On  the  enemy's  mission. 

E:  On  the  accomplishment  of  HIS  mission.  In  the  defense,  1)  what  of  the 
capabilities  that  the  enemy  has  would  do  him  the  most  damage  and  2) ,  the 
harder  part,  what  they're  likely  to  do.  Which  can  have  a  significant  impact 
on  things. 

J:  O.K.,  lets  start  now.  You  didn't  cover  the  issue  of  next  mission,  so, 
don't  go  into  the  detail  yet,  give  me  a  general  summery.  In  other  words,  I 
have  a  mission,  what  am  I  looking  to  do?  In  general,  what  are  you  looking  to 
do? 
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E:  I* a  going  to  tell  him  what  the  eneay  is  going  to  do  both  in  the  near  tern 
and  the  longer  tern,  which  hopefully  aight  be  out  at  48  hours  or  longer  in  a 
Euoropean- type  fast  aoving  warfare.  Or  at  least  out  to  24  hours.  It  could  be 
out  to  two  nonths  in  a  Vietnam  situation,  or  6  months. 

J:  So  you're  looking  to  see  what  the  near  term  and  long  term  threats  are. 

E:  Right. 

J:  How  do  you  determine  what  they  are? 

E:  In  the  near  term,  I  look  at  the  forces,  and  the  terrain  and  weather,  which 
can  have  the  greatest  impact  on  our  ability  to  accomplish  the  mission,  which 
in  these  cases  has  to  be  large,  aggressive,  mobile  heavy  firepower  forces. 

Now  if  we  [he?]  had  the  world's  largest  airforce,  I  would  have  included  that 
in  the  possible  threats,  but  that  wouldn't  throw  us  off  this  ground.  Those 
two  major  forces  there  could  throw  him  off  that  ground. 

J:  So  how  do  you  determine,  O.K.,  so  you've  given  some  criteria;  large, 
aggressive,  mobile,  and  heavy  firepower,  what  criteria  are  you  using  to 
evaluate,  to  determine  that's  a  threat.  So  they  all  impact  into  some  notion 
of  force  ratio. 

E:  And  his  mobility  characteristics,  and  hidden  in  the  force  ratio  are  the 
capabilities  of  his  weapons  systems,  and  leaders  and  all  that. 

J:  O.K. ,  wait  a  second.  One  more  time,  go  through  all  that. 

E:  Well,  in  force  ratios,  his  mobility  capability,  and  hidden  Inside  his 
force  ratios  you'd  like  to  have  some  idea  of  the  relativity  of  weapons 
capability.  In  other  words,  the  normal  way  of  measuring  force  ratio  is  he  has 
three  battalions  and  I  have  one,  that's  three  to  one,  but  he  might  have  a 
weapons  system  that  makes  him  equal  only  one  to  one,  or  make  him  seven  to  one, 
so  there  are  some  qualitative  adjustments. 

J:  So  mobility  and  weapons  capability.  What  else  impacts  on  force  ratio? 

E:  As  I  said,  the  terrain  and  weather  are  always  there.  His  previous 
practices,  is  he  aggressive?  I  mean,  fighting  the  Russians  Is  different  than 
fighting  the  Chilians. 

Craig  Cook:  Fighting  on  home  ground  may  be  different  from  fighting  elsewhere. 
E:  That's  right,  it  all  depends  on  where  you  are  also. 

K:  So  you're  saying  that  all  those  things  are  part  of  force  ratio;  mobility, 
capability,  previous  practices,  and  ... 

E:  Force  ratio  is  a  rule  of  thumb  which  is  generally,  I've  got  so  many 
battalions  and  you've  got  so  many  battalions.  But  then  they  have  to  be 
qualified  by  the  other  factors. 


A-53 


K:  So  you  compute  the  normal  force  ratio  and  then  you  say  "but  he's  mobile" 
and  so  you  make  an  adjustment ,  and  then  you  say  "but  the  terrain  is  such  and 
so"  and  you  make  another  adjustment. 

E:  But  see,  the  commander  has  also  studied  the  terrain,  he  also  knows  the 
forces,  so  he  has  already  got  part  of  that  knowledge  himself.  He  knows  its  a 
totally  mounted  division.  Everybody's  on  tracks  and  wheels,  so  he's  already 
taken  that  in. 

K:  Are  you  going  to  give  the  commander  standard  force  ratio,  or  are  you  going 
to  say  "this  is  a  force  ratio  that  I  have  adjusted  to  take  account  of  weather 
and  terrain  and  mobility  and  all  these  things." 

E:  The  G- 3  comes  up  with  a  force  ratio  and  then  I  give  him  the  red  forces  and 
then  he  takes  his  blue  forces  and  says  "aha.”  And  we  together  say  it’s  seven 
to  one  in  this  area,  two  to  one  in  this  one.  It's  a  joint  project,  but  the 
G-3  is  the  leader  in  that,  providing  the  blue  force  data. 

J :  How  do  you  determine  what  long  term  threats  will  be? 

E:  Size  and  distance  and  time  to  use  or  deployment.  In  other  words,  distance 
from  the  front  in  this  case.  And  anything  else  you  know  about  their 
capability  to  initiate  an  attack.  For  example,  suppose  they've  been  in  the 
combat  for  two  weeks  and  been  withdrawn,  they  may  not  be  ready  to  attack 
tomorrow,  but  a  week  from  now  they  may  be  in  a  much  better  . . . 

J:  How  do  you  determine  if  an  enemy  is  sufficient  size  to  pose  a  threat  to 
you?  How  do  you  relate  the  magnitude  of  threat  to  the  size  of  the  enemy 
force? 

E:  From  the  G-2's  point  of  view,  I  know  that  he  [enemy?]  has  a  doctrine  of 
how  much  of  a  force  ratio  advantage  he  would  like  to  have  before  he  attacks  in 
artillery,  in  tanks,  in  manpower,  in  divisions,  in  aircraft.  I  know  that's 
been  the  historical  way  of  doing  it.  What  I'm  not  sure  is  would  it  be  in  this 
situation.  I'm  starting  off  on  the  premise  that  it  is.  I  also  know  that  we 
think  we  can  defend  the  three  to  one  ratio  successfully  and  so  therefore  we 
know  he  likes  to  have  say  six  to  one  In  the  breakthrough  areas,  so  based  on  my 
education  on  his  doctrine,  I  would  come  up  with  that.  Can  I  make  a  comment 
and  interrupt? 

J :  Sure . 

E:  I  encourage  us  to  keep  our  eyes  out  to  see  whether  or  not,  like  in  all  the 
computer  workstations  that  are  now  being  introduced  to  the  Army,  could  we  help 
them  in  how  to  build  the  knowledge  that  should  be  in  there,  which  is  sort  of 
the  semi* factual  knowledge  and  the  accepted  rules  of  thumb,  six  to  one,  you 
know,  ratios.  The  way  they've  done  it  is  to  take  what  has  been  done  manually 
and  try  to  put  It  on  a  computer.  I  would  just  suggest  we  keep  our  eyes  open 
to  see  whether  through  elicitation  methodology  we  could  help  them  come  up  with 
a  better  way  of  putting  this  on.  That’s  kind  of  a  wild  hope.  I’m  not  trying 
to  press  that  idea  too  far,  but  I  would  say  we  ought  to  keep  our  eyes  open, 
because  there's  a  little  bit  of  reluctance  to  accept  the  workstation  solution 
as  being  a  good  way  to  train  that  young  captain  in  the  future. 
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R:  I'd  suspect  that  this  will  be  the  subject  o£  another  session.  Inside  the 
total  expertise,  there  are  specific  places  where  there's  some  promise  for 
computerized  expert  systems,  but  1  think  that's  not  today.  I  don't  want  to 
interrupt  John's  flow,  unless  in  the  course  of  this  particular  scenario,  it 
happens  to  dawn  on  you  that,  hey,  here's  a  place  where  it  would  be  promising 
to  have  expertise  computerized,  but  I  think  we  should  let  John  run  his  course. 

J:  How  do  you  determine  what  distance  the  enemy  has  to  be  at  before  it's  a 
threat? 

E:  Again,  that's  an  examination  of  a  road  map  and  terrain  and  a  straight 
computation  of  how  long  it  would  take  them  to  make  the  move,  added  on  top  of 
his  normal  doctrine.  In  other  words,  if  you  accept,  for  example  that  maybe 
the  second  echelon  armies  will  sit  off  at  200  km,  when  they  start  to  move 
forward  and  you  see  they're  inside  of  100  km,  and  maybe  at  50  km,  then  you 
begin  to  think  that's  an  important  threat.  The  road  map  then  does  not  permit 
an  easy  diversion  left  or  right  off  a  great  distance,  and  they  might  not  be 
attacking  you,  but  it  would  appear  that  they  would  come  in  and  attack  the 
division  on  your  flank.  So  it's  distance  and  location,  and  the  terrain  that 
supports  it. 

J:  In  other  ^ords  whether  they  can  get  at  you. 

E:  Yeah.  Very  simple. 

J:  O.K. .  time  issue? 

E:  There's  the  standard  rates  of  march,  standard  condition  of  roads,  you  can 
compute  the  time  that  it  will  take  them  to  make  the  move. 

J:  Now,  how  do  you  determine  how  much  time  is  a  threat  for  you,  that  the 
enemy  could  get  there.  How  much  time  are  you  going  to  worry  about? 

E:  It's  taught  in  the  book.  The  division  commander  thinks  out  24  to  48 
hours,  I  think  the  corps  commander  thinks  out  72  to  96  hours. 

J:  So  that's  what  you're  saying  ... 

E:  He  tries  to  plan  ahead  that  far.  At  the  G-2  I  should  be  trying  to  get  out 
the  48  if  at  all  possible.  I've  got  to  worry  about  right  now,  but  I  should  be 
doing  part  of  my  thinking  thinking  out  further  ahead. 

J:  O.K. ,  so  that's  doctrine  based,  what  you're  saying.  O.K.,  and  capability 
to  initiate,  how  do  you  decide  what  their  capability  is  to  initiate? 

E:  I'm  sorry,  the  capabilities  to  initiate  ...? 

J:  The  attack. 

E:  Again,  I  think  the  quick  rule  of  thumb  is  the  force  ratios. 

J:  So,  where  they  are  out  there? 
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E:  Veil,  you  first  of  all  have  some  indication  of  what  they've  been  doing. 

If  they've  been  attacking,  and  they've  got  aore  forces  moving  up,  logically 
you  would  expect  thea  to  continue  the  attack. 

J:  Well,  these  are  long  term  guys,  so  they're  not  attacking. 

E:  Well,  maybe  they've  been  attacking  you,  but  you've  been  able  to  hold  them. 
Now  there's  a  reinforcing  force  coming  up,  a  reserve,  second  echelon,  who  will 
continue  the  attack,  and  he  changes  the  force  ratio  equation  from  2  to  1  in 
front  of  you  to  7  to  1  in  part  of  your  front. 

J:  So,  it's  the  reserve  plus  the  guys  on  line. 

E:  Yeah. 

J:  Let's  look  at  this  example  here,  which  is  one  you're  familiar  with.  All 
right,  so,  I'd  like  you  to  make  some  sort  of  assessment  now  of  what  the 
threats  are  to  the  52  mech  division. 

E:  For  background  I'm  assuming  the  head  of  the  column  ...  what  time  is  that? 
J:  6:00  A.M. 

E:  So  at  6:00  A.M.  the  head  of  the  column  is  55,  in  other  words  I  have 
instantaneous  information,  or  was  that  midnight  last  night? 

J:  No,  that's  right  now. 

E:  Do  I  know  that  right  now?  I'm  asking,  there  is  a  time  delay,  and  I  want 

to  make  sure  I  recognize  that.  A  pilot  airborne  saw  them  moving  forward  with 

their  lights  on,  saw  a  column  100  km  long  and  reported  it. 

J :  For  now  it's... 

E:  Flights  report  in  through  the  air  liaison  officer  to  your  CP  and  he  walked 
in  and  said,  "one  minute  ago,"  (the  reason  I  say  this  is  there's  normally  a 

time  limit) ,  "one  minute  ago  our  four  pilots  saw  a  column  of  vehicles  with 

their  lights  on  appeared  to  be  more  than  100  km  long" . 

J:  Assume  the  information  is  100%  accurate.  God  just  told  it  to  you. 

E:  O.K. 

(laughter) 

K:  It's  good  that  we  have  God  on  our  side. 

J:  No,  he's  just  flying  an  aircraft  up  there. 

R:  What  are  the  markings  on  the  aircraft? 

J :  Halos  and  wings . 
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E:  O.K. ,  first  I  would  calculate  the  worst  case,  assuming  that  they  have 
refueled  and  had  their  orders  and  knew  what  they  were  doing  and  required  no 
further  topping  off  of  supplies  in  any  way,  particularly  fuel... 

J:  Wait,  hold  on  a  second... 

E:  I  would  probably  just  calculate  the  time  looking  at  the  road  net,  I  would 
calculate  the  first  place  at  which  they  could  attack  in  strength  assuming  they 
continue  their  march. 

J:  O.K. ,  so  time,  now  you  have  an  armored  division  as  your  lead  element. 
What's  it  going  to  do?  How  quick  can  it  get  to  you? 

E:  50  km?  Soon,  soon.  With  the  round  about,  it  makes  It  80  km,  at  say  20  km 
per  hour,  average  speed,  they  might  be  here  in  four  hours.  They  might  be  at 
the  point  of  contact  in  four  hours.  That's  worst  case.  They  would  probably 
have  to  break  into  some  sort  of  combat  formations.  If  they  were  going  to  coil 
up  and  top  off  their  tanks  before  hand,  8  to  12  hours,  or  they  could  take 
advantage,  this  is  6:00  in  the  morning,  they  might  take  advantage  and  attack 
tomorrow  morning,  let's  see,  this  is  August,  daylight's  about  3:30?  4:00?  Ah, 
a  little  later,  first  light  tomorrow  morning.  Make  a  passage  of  lines  during 
the  dark.  So  you  know,  they  could  attacl  late  afternoon  if  they  just  moved 
and  attacked,  or  they  would  have  time  to  cc  .  up  and  make  a  passage  of  lines 
in  the  nighttime  and  attack  at  first  light. 

J :  Why  do  you  think  they  might  wait  all  night? 

E:  To  give  them  more  time  for  reconnaissance,  though  it  would  give  them 
time...  The  reason  I  think  they  would  attack  at  daylight  rather  than  during 
the  night  is  that  your  bringing  people  that  have  never  seen  the  battlefield 
forward,  making  a  passage  of  lines  after  dark,  conducting  a  night  attack  on 
strange  terrain,  and  having  to  go  from  column  Into  attack  formations,  and 
that's  difficult.  If  you  gave  them  more  time  to  move  their  forces  up,  to  do 
the  last  minute  reconnaissance,  to  have  a  good  artillery  preparation,  and  jump 
off  at  first  light,  they  could  have  air  support  during  daylight.  They  could 
have  better  observed  artillery  support  during  daylight  and  their  direct  fire 
weapons  would  be  more  effective.  So,  they  would  get  their  firepower  together 
better  in  daylight,  they  would  get  the  advantage  of  surprise  if  they  did  it  at 
night. 

R:  Excuse  me,  I  need  to  interrupt.  I'll  have  to  leave  in  a  few  minutes.  I 
just  wanted  to  make  sure  a  couple  of  things  were  covered,  but  I  see  no  reason 
why  you  shouldn't  continue  after  I  leave,  It  sounds  like  you  have  a  productive 
dialogue  here.  But  while  everybody  is  here,  let  me  repeat  what  I  said  before, 
that  I  very  much  appreciate  any  comments  on  what  you  think  you  learned  from 
this  experience,  at  any  level  you  think  might  be  relevant  to  the  project.  I'd 
suggest  probably  a  memo  to  John,  with  a  copy  to  me,  and  whoever  else  you  think 
is  necessary.  Is  there  anything  that  anybody  wants  to  raise  now?  I  see  no 
particular  advantage  in  raising  it  now,  except  that  if  there  is  something  that 
anybody  wants  to  raise  or  question  that's  best  done  when  we're  all  here,  I'd 
like  there  to  be  an  opportunity  to  pick  it  up.  O.K,  if  not.  then  I  will 
excuse  myself.  I  think  your  pursuing  this  as  you  see  fit  makes  good  sense. 

Do  you  think  it's  important  for  the  rest  of  the  group?  Well  I  suppose  the 
rest  of  the  group  can  decide  if  they  want  to  stay. 
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E:  I'ia  wondering  if  this  sort  of  one  on  one  [isn't  the  best  way  to  proceed]. 

R:  That's  right,  but  the  assembled  gathering  can  decide  if  they  find  it 
helpful  to  eavesdrop  on  this.  1  need  to  excuse  myself.  Thank  you  very  much, 
Ennis . 

J:  Due  to  practical  constraints,  I  don't  want  to  consider  this  too  much 
longer . 

Craig  Cook:  I'd  agree  with  that. 

E:  I  was  hoping  we  would  end  by  2:30. 

J:  Oh,  even  before  that. 

E:  Good. 

J:  O.K.,  when  last  we  left  our  heroes...,  you  were  saying  that  there's  better 
preparation  if  they  attack  the  next  morning,  but  more  surprise  at  night  (2). 

E:  For  example,  if  the  front  line  attacks  were  being  successful  and  they  were 
actually  moving  our  forces  back,  they  might  tell  them  to  continue  the  attack 
in  the  nighttime  because  they  really  aren't  meeting  resistance. 

J:  Is  that  because,  then,  the  added  advantage  of  the  better  air  support  and 
artillery  support  and  more  familiarity  with  the  terrain  would  not  be 
necessary? 

E:  Yes.  The  advantage  of  the  maneuver  would  outweigh  the  advantages  of 
firepower.  Speed  would  be  the  [critical  factor?]. 

Theresa:  Is  that  just  general  work  related  knowledge,  or  is  there  something 
in  Soviet  doctrine  also  would  . . . 

E:  Soviet  doctrine,  U.S.  doctrine,  taught  in  the  schools...  I  mean  there's 
nothing  here  that  the  guy  [doesn't  know?].  One  other  comment.  When  the  guy 
comes  out  of  Fort  Leavenworth  or  out  of  Fort  Huachuca,  in  many  ways  he  is  the 
best  trained  for  the  job  at  that  point.  Then  he  goes  off  and  does  something 
else  for  6  years,  and  then  they  make  him  a  division  G-2  and  he's  trying  to 
remember  what  he  learned  back  in  school.  He  knows  other  things  he  didn't  know 
in  school,  so  the  guy  coming  out  of  school  Is  not  a  novice.  He's  had  good 
basic  training.  He  will  know  this  sort  of  thing.  He  will  do  it  faster  than  I 
will.  And  better  (3). 

J:  You  seem  to  be  invoking  a  new  concept  here  when  you  talk  about  maneuver 
and  air  support,  artillery  support,  and  all  that,  which  are  factors  which 
contribute  to  the  success  of  an  attack.  O.K. ,  would  you,  as  you  know  them, 
say  what  they  are,  what  factors  do  contribute,  other  than,  say,  size  of  force 
and  sophistication  of  weapons  and  things  like  that.  Some  of  these  more,  I 
don't  know  If  you  want  to  call  them  intangibles,  they're  not  intangible. 

E:  O.K.,  we've  now  moved  from  the  G-2  by  himself  to  the  general  knowledge. 
This  is  as  important  to  the  G-3  and  the  commander  as  it  is  the  G-2,  but  ... 
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J:  The  G-2  knows  about  this  ... 

E:  Yeah.  Everybody  should  have  a  common  basis  of  doctrine  to  start  from. 

Let  me  give  you  an  example,  if  I  can  move  a  division  column  40  km  a  night 
without  moving  a  shot  down  a  road,  taking  a  chance  that  the  head  of  the  column 
will  get  stopped  somewhere  and  there  will  be  a  fiasco.  But  if  I  don't  get 
piled  up  and  disorganized  in  darkness,  which  is  very  complex,  if  I  can  somehow 
break  through  any  thin  resistance  and  then  go  60  km,  then  maneuver  means 
everything,  because  if  I  am  60  km  in  his  rear,  then  I  have  turned  and  made 
useless  all  his  defensive  positions.  If  I  don't  think  1  can  make  that 
breakthrough  of  the  crust,  then  I  want  to  get  all  my  firepower  lined  up  and 
get  all  my  troops  organized  to  get  the  maximum  effect  to  breakthrough  and  then 
exploit. 

J:  Now  you've  involved  a  new  concept,  which  is  factors,  or  criteria,  for 
determining  success  of  a  mission,  an  attacking  mission,  which  namely,  breaking 
through  into  the  defenses  rear. 

E:  Well,  my  mission,  as  I  say,  is  the  same,  I  guess  I  was  looking  at  two 
options . 

J:  Well,  you're  talking  about  the  enemy's  mission  now. 

E:  That's  right.  The  enemy's  mission  was  to  seize  crossings  over  the  Rhine. 
This  was  the  army,  the  second  echelon  forces  mission,  to  seize  crossings  over 
the  Rhine,  100  km  in  the  rear.  I  am  the  lead  division.  Soviets  don’t  give 
you  this  flexibility,  but  let's  say  that  I  have  the  option  of  making  a  hasty 
night  attack,  breaking  through  and  then  going  like  the  devil.  Or  I  can  put  my 
forces  in  a  ford  assemble  area,  send  my  leaders  forward,  do  the  final 
reconnaissance,  get  things  lined  up,  get  my  artillery  In  position,  pile  a  lot 
of  artillery  and  firepower  into  it,  and  then  try  to  break  through  at  first 
light  and  then  reach  the  goal.  You're  now  down  in  the  judgment  of  courses  of 
action,  you  see  those  are  just  ...  I've  kind  of  led  us  astray. 

J:  I  think  we'd  better  take  a  time  out  and  summarize  this.  You're  surprising 
me  now,  and  I  don't  know  if  it's  because  we've  been  talking  about  the 
scenario.  I've  got  you  in  a  generative  mode  instead  of  an  interpretative 
mode.  Because  I  would  expect,  just  by  asking  you  what  factors  make  attacks 
strong,  or  what  factors  do  attacks,  you'd  just  read  off  a  list  that  says 
'Field  Manual  X' . 

E:  That's  right,  the  principles  of  war. 

J:  But  now  when  I'm  asking  you  this  question,  you're  describing  in  concrete 
terms  an  example  of  what  would  have  been  a  successful  attack.  I'm  trying  to 
wonder  if  that's  how  you  think  about  it  or  or  if  by  getting  you  in  the 
scenario  I've  got  you  in  a  generative  mode,  so  I've  just  happened  to  put  you 
in  a  piece  of  knowledge  that  now  you're  working  in. 

Marvin:  You're  coming  down  this  hierarchy,  and  I  think  you're  right,  I  mean, 
you  had  him  down  at  this  level,  you  asked  him  a  question  that  might  have  sent 
him  up  there,  but  it  didn't,  that's  exactly  what  I  was  saying. 
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J:  Right,  So  I'm  just  trying  to  see  if  you  have  any  reflections,  if  I've  got 
you  in  scenario  mode  now  because  through  having  talked  about  this  scenario  a 
lot  during  the  questions.  Or  if  that's  the  way  you  normally  think  about,  I 
mean  do  you  think  about  the  objectives  . ..? 

E:  I  guess  I  am,  what's  the  objective,  objective  oriented,  and  I  guess  that's 
sort  of  leading  my  thinking. 

J :  I  mean  . . . 

E:  But  I'm  not  sure,  I  mean,  everybody  thinks  differently,  so  ... 

J:  I  understand,  but  I'm  talking  about  you,  just  you,  I  don't  care  about 
anyone  else. 

Craig  Cook:  You've  also  made  a  few  comments  along  the  way  about  "it's  in  the 
book",  this  is  textbook  kind  of  stuff,  is  it  possible  that  you're  bored  with 
this  topdown  approach ,  that  . . . 

E:  No,  no,  no.  I  was  just  trying  to  indicate  that,  I  guess  I  was  trying  to 
lend  substance  that  what  I'm  saying  is  not  terribly  imaginative,  that  all 
G-2s,  G-?s  and  commanders  ought  to  come  up  with  generally  the  same  response. 
Now  there's  some  who  would  consider  a  set  of  totally  different  factors,  time 
and  space  were  kind  of  the  drivers  for  me  on  the  first  part.  But  you  are 
taught  a  lot  of  things,  you  are  taught  the  rate  at  which  forces  move,  you  are 
taught  to  study  the  terrain  and  road  map,  you  are  taught  to,  you  know,  and 
then  use  it. 

J:  I'm  kind  of  thinking  that,  I  mean  I  could  keep  doing  this,  but  I'm  kind  of 

satisfied  that  I've  a  feel  for  what's  happening,  so  we  could  adjourn. 
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ATTACHMENT  A- 2:  COMMENTS  ON  MICROCOSM  TRANSCRIPT 


These  comments  are  keyed  to  numbers  in  the  text  of  the  transcript  in 
Attachment  A-l  (consecutive  within  each  numbered  section). 

1.0  INTRODUCTION 

1.1  The  ground- rules  for  the  session,  distributed  in  advance,  are 
in  Attachment  A- 3. 

1.2  The  Treysa  Defense  Scenario  is  an  example  of  a  Fulda  gap  scenario  In 
Western  Germany.  The  U.S.  is  defending  against  a  superior  Soviet  force 
threatening  a  communications  center  at  Treysa.  The  subject,  General 
Whitehead,  had  studied  intelligence  summary  and  other  material,  which  had  been 
prepared  (for  another  purpose)  for  a  division  level  G-3  preparing  to  make  a 
recommendation  between  three  courses  of  action  within  a  4 -hour  period. 

2.0  BROAD  GUIDANCE 

2.1  This  refers  to  two  pages  of  notes  EW  had  prepared  ahead  of  time  (see 
Attachment  A- 4) , 

2.2  Knowledge  item:  a  key  G-2  function  is  not  to  be  a  knowledgeable  expert 
but  to  use  people  who  may  be. 

2.3  EW  is  going  straight  to  narrow  scenario-specific  guidance,  though  asked 
to  be  broad  initially.  This  may  suggest  that  his  most  readily  accessible 
knowledge  is  generative,  and  more  than  that,  Lnstantiative ,  i.e.,  expressed  as 
illustrative  instances.  This  is  different  from  a  production  rule,  where  the 
defining  conditions  are  stated  explicitly.  In  a  sense,  EW's  knowledge  here  is 
not  quite  lnstantiative .  The  conditions  being  specified  In  the  pre- session 
documentation  would  only  be  strictly  lnstantiative  if  EW  were  actually  in  the 
real  world  situation,  where  much  of  the  conditioning  information  would  be 
implicit ,  and  experential,  without  being  verbalized.  The  instantiated 
knowledge  could  only  come  out  of  the  expert  responding  to  the  real  situation, 
including  all  the  subtle  cues  that  do  not  get  into  writing.  The  conditioned 
reflex  would  be  an  example,  where  the  subject  may  not  be  conscious  of  any 
rules  generating  the  response . 

It  may,  of  course,  be  that  EW's  total  knowledge  is  interpretative,  but  not 
that  part  of  his  knowledge  which  goes  beyond  that  of  the  trainee  G-2,  e.g.,  it 
does  not  include  the  largely  interpretative  material  taught  in  school  and 
given  in  doctrinal  documents. 

2.4  Knowledge  item  inferred  from  instance:  "check  intelligence  reports  for 

plausibility,  as  generated  by  doctrine  and  prior  judgment.  If  this  generates 
major  uncertainty,  you  need  more  information."  EW  did  not  express  his 
knowledge  in  this  production  rule  form.  I  induced  it.  We  could  have  pressed 
him  to  come  up  with  a  production  rule  or  confirm  one  that  we  inferred.  This 
can  be  raised  in  a  follow-up  elicitation  session  (which  I  do  not  recommend  in 
view  of  the  shifting  emphasis  of  the  project).  Another  implied  production 
role:  intelligence  reports  at  the  beginning  of  the  war  are  suspect,  because 

no  one  is  yet  an  expert. 
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2.5  The  production  rule  here  was  expressed  fairly  directly,  rather  than 
needing  to  be  inferred  from  an  Instance. 

2.6  Knowledge  item  in  form  of  mid-level  rule:  "focus  on  24  hours  ahead  and 
resist  any  inclination  to  report  the  present  or  past  only."  This  is  the  kind 
of  guidance  that  might  go  into  a  doctrinal  document  and  is,  perhaps,  already 
there . 

2.7  Knowledge  Item:  "Expect  attacks  at  boundaries,  because  enemy  thinking 
is  likely  to  be...."  This  knowledge  is  already  in  doctrine,  i.e.,  part  of  the 
knowledge  we  are  eliciting  is  a  recapitulation  of  doctrine.  Possibly  all  of 
the  broad  and  narrow  generalizations  from  EW  which  are  not  specific  to  this 
situation  derive  from  doctrine.  Perhaps  the  loose  structure  of  this  part  of 
the  session  means  we  are  only  getting  at  the  readily  accessible 
generalizations.  Perhaps  we  could  have  pulled  original,  non-doctrinal  general 
knowledge  out  of  EW,  if  our  elicitation  had  been  more  active. 

2.8  Knowledge  Item  which  is  an  instantiation  of  a  more  general,  unstated, 

rule:  "Get  and  "fuse"  many  different  types  of  information  (analyses)."  This 

suggests  that  a  potential  mode  of  knowledge  elicitation  is  for  the  elicitor  to 
construct  a  generalization  (based  on  low  level  rules  or  instantiations)  and 
play  it  back  to  the  subject  for  confirmation  (e.g.,  the  above  inferred 
generalization  about  plural  analysis) . 

2.9  Knowledge  Item,  which  is  a  recommendation,  followed  by  an  explanatory 
principle:  "Find  out  about  the  larger  forces  in  the  rear,  because  it  has  a 
greater  impact  on  our  division."  This  is  a  common  case  in  this  session. 

2.10  A  last- -and  successful- -attempt  by  the  elicitor  to  extract  guidance  not 
specific  to  the  scenario.  Host  of  it  is  stipulation  of  instantive  advice 
given  earlier  (e.g.,  "Speed  is  more  important  than  accuracy.") 

2.11  Reiteration  of  EW,s  early  meta-rule:  "G-2's  task  is  management." 

3.0  SPECIFIC  INTELLIGENCE  AREA:  ASSESSING  ENEMY  MOVEMENTS 

3.1  Pre-session,  EW  had  been  given  a  list  of  half  a  dozen  potential 
intelligence  items  to  focus  on,  including  the  location  of  enemy  command  posts, 
put  forward  by  DSC  staff  with  army  experience.  EW  proposed,  instead,  an  item 
which  he  felt  should  have  higher  priority- -whether  the  9th  Division  is  moving. 

Technical  aside  on  knowledge  elicitation  methodology. 

An  important  kind  of  knowledge,  relevant  to  the  new  thrust  of  the  project 
(intelligence  analysis),  and  quite  likely  to  be  where  the  expert  has  something 
to  offer  over  doctrine,  is  the  interactive  aspects  of  intelligence,  i.e.,  the 
alternation  of  assessment  and  information  gathering  on  any  particular 
requirement;  in  this  case,  the  priority  intelligence  requirement  is  .  ,iemy 
reserve  movement.  There  is  an  initial  assessment- -that  it  is 
stationary- -which  is  evaluated  as  suspect,  based  on  conflict  with  other 
evidence.  This  stimulates  plural  analysis- -going  off  to  quite  different 
sources  of  intelligence,  such  as  corps  mohawk  and  visiting  the  German  Corps. 
The  analysis  needs  to  be  continually  updated,  so  that  the  commander  has  the 
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best  current  estimate  in  time  to  make  relevant  decisions,  e.g.,  where  to  move 
his  own  reserves. 

A  computerized  analysis  aid,  focused  on  fusion,  could  be  sequentially  updated 
for  probabilistic  assessment  of  the  PIR.  In  this  case,  the  standardized 
output  might  consists  of  a  list  of  current  hypotheses  (9th  Division 
stationary,  moving  south,  moving  north),  with  either  probability  numbers  or 
verbal  likelihood  statements.  Immediate  input  could  be  a  list  of  intelligence 
sources  with  some  measure  of  their  diagnostlcity  for  the  target  issue,  e.g.: 
intelligence  summary,  prior  judgment  based  on  doctrinal  assumption, 
surveillance  aircraft  report,  German  report.  A  key  knowledge  elicitation 
issue  is  how  these  input  questions  should  be  posed,  so  as  to  make  best  use  of 
expertise.  This  surely  dominates  the  convenience  of  the  logical  algorithm, 
linking  input  to  output.  Bayesian  updating  would  be  a  natural  algorithm, 
which  produces  output  in  the  right  form.  However,  it  calls  for  prior  judgment 
(in  the  same  form  as  the  output  so  that  it  is  probably  O.K.),  but  also 
likelihood  assessments  (as  a  measure  of  diagnostlcity) . 

In  its  exact  form,  this  requires  assessments  of  the  probability  of  the 
intelligence  report  given  possibly  hypotheses  about  the  target  issue  (e.g., 
PIR).  Perhaps  we  could  elicit  a  measure  of  "surprise,"  in  linguistic  terms, 
such  as  "inconceivable,"  through  "likely"  to  "inevitable.”  The  machine  would 
need  to  translate  from  verbal  input  to  quantitative  Input,  to  quantitative 
output,  to  verbal  output.  The  important  knowledge  elicitation  technique  issue 
is  the  trade-off  between  defining  Input  which  is  easy  to  elicit;  and  input 
from  which  relevant  output  can  most  readily  be  inferred.  For  example,  a 
"persuasiveness”  scale  may  be  easier  to  get  than  a  "surprise"  scale,  to  link 
an  intelligence  report  to  a  hypothesis,  even  though  the  latter  fits  better 
into  the  Bayesian  updating  algorithm. 

Knowledge  to  be  elicited  in  advance  of  the  battle  to  be  aided  can  be  of  two 
sharply  distinct  kinds.  One  is  of  the  expert  G-2  to  enhance  the  local  G-2,  of 
the  type  we  are  eliciting  from  EW.  It  would  be  doctrinal  types  of 
information.  My  intuitive  judgment  is  that  it  would  be  too  difficult  to 
elicit  this  Information  in  advance  of  any  particular  battle  in  a  way  that 
would  be  relevant  after  the  battle  commences  (though  I  suspect  it  will  be  a 
seductive  idea  to  army  bureaucrats!)  The  way  for  doctrine,  and  expertise 
beyond  doctrine,  t  ■>  be  made  usefully  available  I  feel  sure,  is  by  more 
conventional  training  and  briefing  of  the  G-2  and/or  his  staff.  Not 
computerized. 

However,  a  generic  fusion  aid  based  on  analytic  knowledge  elicitation  seems 
quite  promising.  However,  the  substantive  knowledge  to  feed  it  will  always 
need  to  be  tailored  to  the  local  topical  situation  and  I  am  not  optimistic 
that  that  part  can  be  computerized. 

A  more  ambitious  extension  to  this  fusion  aid  would  permit  projecting  the 
value  of  intelligence  collection  options.  The  underlying  algorithm  might  be 
decomposed  error  analysis  if  the  target  issue  is  itself  quantitative,  e.g., 
how  long  until  enemy  attacks?  In  this  extended  aid,  the  diagnostlcity  of  the 
intelligence  needs  to  be  assessed  before  it  is  received.  It  would  need  to  be 
a  more  readily  interpretable  version  of  the  mathematical  concept  of  "the  prior 
expectation  of  the  posterior  credible  interval." 
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The  above  discussion  all  has  to  do  with  the  use  rather  than  the  elicitation  of 
the  knowledge,  but  I  do  not  think  they  can  be  usefully  separated.  You  must 
determine  what  knowledge  is  to  be  elicited,  and  in  what  form,  before  you  can 
meaningfully  address  the  appropriate  way  to  elicit  it. 

3.2  Locatic  of  the  Command  Post  is  in  an  inference  hierarchy.  It  has  some 
interest  in  its  own  right  (so  it  can  be  killed) .  It  is  also  diagnostic  of 
another  IR:  the  location  of  the  9th  Division.  In  principle,  we  could  design 
an  expert  aid  which  would  assign  priority  to  IR's  based  on  a  "value  of 
information"  algorithm.  I  don't  believe  one  can  anticipate  enough  of  the 
specifics  of  the  problem  to  do  much  of  that  in  advance. 

This  supports  the  idea  that  intelligence  analysis  can  be  more  readily 
computerized  than  intelligence  collection  management  (since  the  latter 
requires  more  considerations  to  be  taken  into  account) . 

If  the  knowledge  that  is  to  be  elicited  comes  from  developments  in  battle  and 
at  short  notice,  what's  a  significant  constraint  on  the  knowledge  elicitation 
technique.  Elicitation  may  itself  need  to  be  computerized,  since  there  is  not 
likely  to  be  an  elicitation  specialist  at  hand.  In  addition,  the  subject  may 
have  to  be  trained  so  that  he  can  respond  to  the  elicitation. 

3.3  The  influence  diagram  program  (e.g.,  as  developed  by  Ross  Schachter  of 
Stanford)  might  be  a  promising  vehicle  for  the  kind  of  inference  linkage  in 
the  knowledge  elicited  here.  Unless  there  are  certain  very  standardized  kinds 
of  inference  which  occur  often  enough  and  in  a  consistent  enough  form  there 
may  not  be  much  scope  for  pre -programming,  beyond  supplying  a  generic  tool 
which  must  be  adapted  to  the  specific  circumstances  in  the  field.  This  issue 
of  pre-programmed  decision  aids  was  the  subject  of  a  substantial  Navy  research 
effort  in  the  70' s,  funded  by  ONR  as  the  Operational  Decision  Aids  program. 
(Brown,  et  al. ,  1974,  1975). 

One  of  the  rare  situations  to  justify  pre -programming  was  assessing  range  to 
the  target  (Bromage,  et  al.,  1983).  It  is  not  clear  that  there  is  anything  of 
comparable  promise  in  a  battle  combat  situation,  where  the  target  ranging 
problem  is  less  dominating  than  in  submarines  and  is  subject  to  less 
standardized  sources  of  intelligence. 

3.4  This  was  an  attempt  to  get  at  a  deductive  mode  of  thinking  underlying 
the  intelligence  prioritization.  The  conventional  value  of  information 
paradigm  was  clearly  not  near  the  surface  of  EW's  mind.  This  does  not  mean 
that  we  could  not  elicit  knowledge  in  that  form;  only  that  he  was  not  inclined 
to  volunteer  it  in  that  form.  A  useful  follow-on  exercise  would  be  to  conduct 
a  structured  deductive  knowledge  elicitation,  compare  its  implications  with 
subject’s  direct  recommendation.  We  went  part  of  the  way,  by  having  EW 
discuss  the  impact  on  the  three  courses  of  action,  if  his  intelligence 
collection  comes  out  different  ways.  This  would  have  implications  for  how 
important  the  collection  activity  is,  which  may  not  coincide  with  his  direct 
assessment. 

The  type  of  knowledge  elicitation  we  are  exploring  here  does  not  lend  itself 
to  computerization,  and  therefore  is  probably  not  worth  pursuing  on  this 
project. 
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3.5  Another  kind  of  computerized  knowledge  is  a  directed  search  of  agenda 
items,  i.e.,  what  issues  to  look  at  in  certain  circumstances .  This  would  be  a 
kind  of  interactive  exploration  of  a  doctrinal  data  base. 

A  computerized  training  manual  would  be  a  candidate  end  use  for  knowledge 
elicitation,  but  possibly  out  of  the  scope  for  this  project. 

3.6  In  the  normal  course  of  apprenticeship,  the  trainee  elicits  directly 
from  the  expert  by  watching  and  presumably  asking.  No  scope  for 
computerization  here. 

3.7  Most  direct  form  of  inductive  learning  is  watching  the  expert  in  a  real 
situation.  However, EW  believes  that  commentary  by  the  expert  is  helpful  too. 

A  version  of  the  knowledge  transmittal  cycle,  i.e.,  knowledge 
elicitation- -presentation- -use,  is  directly  between  expert  and  trainee  in  a 
training  situation.  Low  priority  now  on  this  project. 

3.8  A  significant  class  of  knowledge  is  how  to  evaluate  an  item  of 
intelligence.  Trainee's  knowledge  base  includes  generic  data  on  accuracy  of 
different  types  of  collector  e.g.,  "most  cannot  see  more  than  five  km".  Scope 
for  a  computerized  reference  book,  as  an  intelligence  analysis  aid,  and 
intelligence  collection  aid.  Not  of  great  interest  to  this  project. 

3.9  Knowledge  item:  example  of  low  confidence  intelligence,  Implying  needs 
to  dig  deeper  with  more  extensive  collection. 

3.10  Knowledge  item:  how  to  assign  confidence  to  source  of  intelligence. 
Generic  rules  of  thumb  on  quality  of  intelligence  reports. 

3.11  Instantiative  reasoning  that  leads  to  conclusion  that  the  9th  division 
is  in  a  certain  place.  Different  indicators  at  different  levels  in  an 
inference  hierarchy  converge,  e.g.,  a  mix  of  forces  characteristic  of  a 
certain  kind  of  division.  What  constitutes  conclusive  evidence  neatly  fits 
into  paradigm  of  hierarchical  inference. 

3.12  One  way  to  elicit  knowledge,  of  substantial  practical  importance,  but 
probably  not  what  we  are  now  after,  is  interactive  elicitation.  The  user  of 
the  knowledge  interrogates  database  (human  or  computerized).  Essentially, 
each  question  is  prompted  by  the  answers  to  previous  questions.  Just  what  is 
going  on  here:  we  dig  into  what  is  behind  EW's  conclusions,  with  selective 
probings.  DSC  has  been  developing  computerized  prof- types  of  this  (Ulvila  and 
Thompson,  1985;  Brown,  1986).  A  computerized  v'"  requires  that  all 
potential  information  to  be  called  up  must  be  .^ogrammed.  This  may  make 
sense  In  some  kinds  of  repetitive  or  large-scale  single  decisions,  which  can 
justify  the  set-up  cost,  but  probably  not  division-level  battlefield 
decisions . 

3.13  Type  of  knowledge:  inference  about  nuclear  weapons.  A  good  example 
where  established  doctrine  is  lacking  because  it  is  new  and  sensitive,  and 
where  eliciting  expert's  knowledge,  is  therefore,  of  great  importance.  For 
much  the  same  reasons,  expertise  is  going  to  be  very  scarce- -and  will  be  of 
the  deductive,  not  inductive,  kind  (i.e.,  there  will  be  almost  no  experential 
knowledge  or  even  reasonable  surrogates  for  it) .  Could  be  in  an  interpretive 
form,  including  the  imposed  interpretative  form  of  a  structure  such  as  PDA. 
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This  is  an  exaaple  where  type  of  knowledge  is  associated  with  knowledge 
structure  and  derivatively  method  of  elicitation.  One  might  expect  change 
during  the  course  of  a  war,  from  deductive  to  inductive  as  doctrine  is 
replaced  by  experience. 

3.14  Type  of  knowledge:  principle  of  war. 

3.15  Here  we  have  extracted  a  complete  PDA-like  chain  of  reasoning,  i.e.,  he 
explicates  why  enemy  movement  information  is  important  to  get  intelligence  on; 
in  terms  of  its  impact  on  commander's  actions,  i.e.,  the  intelligence  is 
important  because  it  tells  him  what  to  do,  but  he  stops  short  of  considering 
the  cost  of  errors . 

Implication  for  knowledge  elicitation  on  action  decisions:  advice  bearing  on 
collection  management  decisions  can  be  specified  at  different  levels  of 
generality:  send  someone  to  talk  to  the  23rd  Panzer  Division;  assess  their 
intelligence  on  the  reserve  movements ;  get  German  information  on  enemy 
movements’,  get  information  on  enemy  movements;  find  out  about  the  most  serious 
threat;  get  information  where  the  cost  of  error  is  greatest;  implement  this 
V01  paradigm  of  PDA.  Each  step  could  be  inferred  by  implementing  next  one 
along. 

Issues  relevant  to  knowledge  elicitation  are:  How  much  of  a  total  reasoning 
chain  is  the  subject  applying?  Can  he  supply  deep  levels  of  it?  Should  the 
knowledge  elicitation  force  him  to?  What  if  the  subject  is  incoherent  (i.e., 
deriving  the  advice  at  different  levels  and  deducing  implications)?  One  major 
mode  of  knowledge  elicitation  to  enhance  the  expert's  knowledge,  by  giving  it 
logical  support  (e.g.,  take  it  into  PDA  and/or  plural  analysis  framework)? 
Alternatively,  knowledge  can  be  selectively  distilled  from  the  expert,  which, 
on  some  grounds  or  another,  is  judged  to  be  most  valuable,  i.e.,  you  may  not 
adopt  his  advice  which  is  based  on  he  is  not  an  expert  in.  He  says:  "Send 
S-2  to  Jones  to  talk  to  the  Germans."  His  expertise  tells  him  that 
complementary  knowledge  about  enemy  movements  is  of  high  value,  which  is  also 
based  here  on  the  "knowledge"  that  the  Germans  will  have  valuable  information 
which  they  are  prepared  to  share.  Somebody  else  may  be  the  expert  to  ask  on 
that  aspect.  Selective  elicitation  of  knowledge  is  helped  by  having  an 
inference  structure  (e.g.,  PDA)  provided  by  the  knowledge  engineer,  not  by  any 
experts . 

3.16  Illustrates  the  importance  of  considering  the  whole  "knowledge  transfer" 
cycle  when  designing  any  part  of  it,  such  as  knowledge  elicitation.  This 
raises  the  question  of  having  the  knowledge  represented  at  the  elicitation 
session  itself,  e.g.,  a  video  tape  of  it,  possibly  with  commentary  added 
afterwards.  The  knowledge  elicitation  for  that  will  be  quite  different  from 
one  where  the  knowledge  is  to  be  transmitted  through  a  computer  program.*  A 
knowledge  elicitation  needs  to  be  considered  in  context. 

4.0  OPEN  DISCUSSION 

4.1  Selection  of  knowledge  to  elicit  is  an  i’-^ortant  part  of  knowledge 
elicitation.  It  requires  substance  expert  as  well  as  a  knowledge  engineering 
expert  to  design  the  strategy. 
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4.2  Specifying  indicators  (e.g.,  Uniting  features  of  the  terrain)  helps 
assess  target  facts  to  neet  intelligence  requirements  (e.g.,  direction  of 
enemy  movement) . 

4.3  More  prompting  of  the  value  of  information  paradigm. 

4.4  Dynamic  updating  of  information.  Major  type  of  knowledge  missing  in  the 
training  database  appears  to  be  the  evaluation  of  multiple  collectors  whose 
output  will  be  fused.  We  learn  Individual  collector  capabilities  but  not  how 
to  put  it  all  together.  (True?) 

4.5  Part  of  analysis  is  adapting  intelligence  to  the  specific  requirements 
of  this  commander. 

4.6  Generic  knowledge:  Don't  get  distracted  by  detail,  except  to  check  on 
its  quality  by  spot  check. 

4.7  The  importance  of  knowledge  to  be  elicited  from  experts  that  goes  beyond 
regular  training.  EW  believes  there  is  substantial  value.  It  has  bearing  on 
what  kind  of  knowledge 'based  aid  would  be  developed.  It  bears  on  how  much  of 
the  aid  replicates  the  training  base,  as  against  what's  missing  from  it,  or 
the  union  of  the  two.  The  implication  might  be  to  augment  training  with 
additional  material,  acknowledging  that  it  might  need  to  be  in  a  rather 
different  form,  (e.g.,  video  tape  of  a  session  like  this  one). 

4.8  Assessing  uncertainty  is  a  major  priority  of  G-2.  When  looking  ahead, 
how  should  uncertainty  be  handled  and  communicated  in  an  intelligence  analysis 
aid? 

4.9  Restating  and  interpreting  the  principles  of  war. 

4.10  Critical  importance  of  fast  estimate,  even  if  poor.  Part  of  the 
analysis  problem  is  when  to  report. 

4.11  On  the  dividing  line  between  collection  and  analysis  is  what  to  look  at 
or  who  to  ask,  which  is  basically  already  within  reach?  (e.g.,  asking  the 
brigades  if  they  see  any  signs  of  reinforcements). 

5.0  STRUCTURED  ELICITATION:  STEP -THROUGH  SIMULATION 

5.1  The  ultimate  function  of  step- through  simulation  here  is  not  quite 
clear.  Assuming  a  particular  U.S.  course  of  action,  we  could  be  addressing 
the  probable  consequences  of  the  action,  which  is  part  of  the  intelligence 
analysis  function.  A  potential  analysis  aid  here  would  be  based  on  knowledge 
from  an  analysis  expert  how  to  do  this  kind  of  step- through  simulation.  This 
is  not  something  you  would  likely  do  on  the  division  level.  Requires  too  much 
effort,  time,  and  expertise.  Step-through  could  also  be  used  for  training,  to 
provide  illustrative  intel  analyses  as  the  sequence  unfolds.  It  Is  the 
surrogate  for  watching  a  senior  perform  In  real  battle. 

5.2  Much  of  the  art  of  intel  analysis  is  the  beaming  of  attention. 

Attention  itself  is  a  scarce  resource. 
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5.3  Force  ratios  are  an  example  of  Intelligence  requirements  whose  analysis 
could  perhaps  be  pre-programmed.  However,  it  is  unlikely  that  the 
diagnosticity  of  the  indicators  can  be  set  much  in  advance,  but  that  would  be 
something  to  check  with  military  experts. 

5.4  Information  seems  so  complex,  rich  and  situation-specific,  that  it  will 
appear  that  the  only  generalization  possible  has  to  do  with  underlying  logical 
structures  for  making  decisions  or  inferences  (which  have  a  function  we  are 
eliciting  knowledge  for).  This  argues  strongly  for  imposing  a  PDA  structure. 

5.5  Sequences  that  get  selected  by  step-through  should  capture  not  only  what 

is  most  probable,  what  is  most  important.  This  argues  against  random  sampling 

at  each  node  in  the  usual  way,  which  may  miss  the  more  interesting,  if  rare, 

sequences.  The  analysis  will  not  be  distorted,  provided  each  simulation  trial 
has  the  appropriate  probability  weighting,  when  not  equal,  as  in  conventional 
simulation.  This  is  an  area  for  methodology  development. 

5.6  Some  unfolding  of  data  in  the  step- through  sequence  casts  light  on  the 
quality  of  an  earlier  collection  management  decision  in  the  sequence  e.g., 
Misjudging  who  is  attacking  and  where  led  to  surveillance  helicopter  forces 
being  wasted,  i.e.,  there  is  less  flying  time  now  available  for  precious 
purposes . 

5.7  The  "feel"  of  human  collectors  (as  opposed  to  their  facts),  is  an 

important  item  of  intelligence  to  analyze.  How  do  you  measure  that  feel? 

Letting  them  talk  in  their  own  terms;  or  by  extracting  answers  in  a  preset  (or 

perhaps  unnatural)  format.  For  example,  posterior  probabilities,  as  against 
prior  and  likelihood  functions.  A  computerized  aid  for  eliciting  this  feel 
would  be  promising. 

6.0  PROCEDURAL  INSIGHTS 

6.1  Elicitor  is  clearly  having  difficulty  getting  at  what  the  expert  thinks 
is  knowledge  incremental  to  the  trainee.  This  is  what  we  are  supposed  to  be 
eliciting  in  this  case.  As  a  result,  we  are  probably  tapping  into  his  total 
expertise,  not  his  incremental  expertise  over  the  trainee.  This  is  probably 
alright.  Important  issue  of  how  finely  defined  a  given  situation  needs  to  be 
(e.g.,  in  a  step-through  sequence)  in  order  to  elicit  the  kind  of  knowledge  we 
are  really  after  (e.g.,  the  increment  between  an  expert  and  a  trainee). 

6.2  One  way  to  capture  the  detail  that  the  G-2  expert  would  have  is  to  make 
sure  we  have  another  resource  expert  who  can  both  supply  what  he  (the  resource 
expert)  believes  the  G-2  expert  would  know  and  also  anticipate  the  flow  of 
questions  and  answers  realistically  (including  responding  "we  don't  have  that 
information" --that  intelligence  is  not  In  his  "dream").  Clearly,  It  Is  not 
going  to  be  possible  to  Monte  Carlo  all  the  possible  branchings  at  this  level 
of  definition.  It  is  probably  alright  if  part  of  the  sequence  is  the  resource 
expert's  modal  enrichment  of  other  parts  which  are  selected  probabilistically. 
Another  case  where  the  technique  of  step -through  needs  developing. 

6.3  We  were  not  able  to  identify  any  specific  place  where  the  expert  had 
knowledge  that  the  trainee  lacked,  at  least  which  would  have  made  a  tangible 
difference.  It  may  not  be  significant  that  it  was  not  articulated.  It  may 


A-68 


still  be  there.  A  major  part  of  the  learning  is  not  knowledge  but  patterns  of 
behavior,  e.g.,  working  the  system  harder  than  you  were  trained  to. 

6.4  We  are  really  ending  up  with  an  annotated  case  study  which  is 
hypothetical.  Not  a  good  test  of  step- through. 

6.5  A  map  exercise  with  interruptions  might  be  a  better  case  study  approach 
to  learning,  but  still  not  quite  the  same  as  the  Harvard  case  study  method. 

It  is  addressing  a  different  purpose  from  step -through. 

6.6  Whether  in  the  step- through  or  case  study  mode,  military  expertise  is 
needed  for  three  roles:  the  expert  whose  knowledge  is  being  elicited,  the 
source  of  scenario  data,  and  the  elicitor.  Lack  of  expertise  impaired  our 
ability  to  do  the  latter.  If  we  are  trying  to  determine  the  "expert's 
increment"  the  suggestion  is  run  the  exercise  twice,  once  with  a  trainee  and 
once  with  an  expert,  and  examine  the  difference. 

6.7  It  seems  clear  that  the  kind  of  knowledge  we  are  going  after  here  is  not 
susceptible  to  machine  coding,  and  therefore  should  probably  not  be  pursued. 

EW  is  confirming  this.  What  can  be  handled  at  a  computer  workstation  is 
probably  low  level  data  or  perhaps  doctrine  retrieval,  rather  than  the  more 
sophisticated  interpretation  we  are  getting  at  here. 

6.8  If  a  suitable  scenario  exists,  it  would  serve  training  purposes,  but  it 
is  not  clear  if  this  is  relevant  to  generating  a  computerizable  product,  which 
now  seems  called  for. 

6.9  More  generally.  It  raises  the  question  which  is  relevant  to  the  project: 
how  is  the  expert's  knowledge  structured?  In  particular,  is  it  high  or  low 
level  rules?  Marvin  is  suggesting,  I  think,  that  it  could  be  tested 
empirically.  If  it  is  low  level,  you  will  produce  it  quickly;  but  may  not  be 
able  to  adapt  so  readily  to  a  new  situation.  I  read  the  evidence  in  this 
elicitation  session  as  indicating  rather  low  level  rules.  Perhaps  that  is 
because  we  are  in  a  situation  familiar  to  him.  In  a  more  unusual  situation, 
e.g.,  nuclear,  he  might  call  up  higher  level  rules.  More  than  one  structure 
may  exist  in  his  head  to  be  accessed  in  different  circumstances. 

6.10  An  important  suggestion  that  we  are  getting  some  high  level  rules, 
because  we  don't  have  a  current  expert,  one  whose  knowledge  we  really  want  to 
elicit. 

6.11  If  we  are  using  step- through  as  a  training  case  study,  everything  except 
specific  function  being  trained  (G-2)  is  treated  as  "  the  outside  world," 
e.g.,  It  includes  the  actions  of  underlings  he  has  to  implement  his 
directives . 

6.12  Important  distinction  between  the  knowledge  of  a  teacher  and  the 
knowledge  of  a  doer. 

7.0  KNOWLEDGE  STRUCTURE 

7.1  The  knowledge  structure  John  Leddo  is  eliciting  appears  to  be  a  top-down 
structure,  with  mission  accomplishment  at  the  top.  (He  developed  a  structure 
on  paper,  as  he  proceeded.)  I  suppose  it  would  generalize  into  some  kind  of 


A-69 


influence  diagram.  This  is  clearly  a  relevant  kind  of  knowledge  which  can  be 
largely  decoupled  from  specific  inference  and  decision  tasks.  Its  value  will 
also  persist  (at  least  its  structure)  froa  day  to  day  and  perhaps  battle  to 
battle . 

A  possible  coaputerized  function,  involving  this  knowledge,  would  be  a 
probabilistic  model  of  key  elements  in  the  battlefield,  which  have  default 
values  which  can  be  overridden  and  updated.  Particularly  as  you  are  unlikely 
to  have  a  methodological  expert  to  work  the  system,  at  division  level.  The 
place  to  start  this  sort  of  thing  is  surely  back  in  the  Pentagon,  rather  than 
out  in  the  field. 

Another  possibility  would  be  a  computerized  piece  of  doctrine  perhaps 
specialized  to  a  class  of  situations  like  the  Fulda  gap.  The  division  will 
probably  not  have  the  necessary  resources  or  the  elapsed  time  to  use  such  an 
aid. 

7.2  We  are  picking  up  different  bits  of  war  lore  illustrated  in  the  context 
of  Treysa  defense. 

7.3  On  its  face  value ,  EW  appears  to  be  saying  that  in  advan  >  of  World  War 
III  starting,  almost  everything  worth  knowing  is  in  doctrine,  i.e.,  there  is 
no  significant  real  world  experience  beyond  that. 
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ATTACHMENT  A-3:  SESSION  GROUNDRULES 

MEMORANDUM 


TO: 

FROM: 

SUBJECT: 

DATE: 


Knowledge  Elicitation  Project  Team 
Rex  V.  Brown 

Elicitation  Working  Session 
November  21,  1986 


GROUND  RULES  FOR  MONDAY’S  WORKING  SESSION 


The  Elicitation  Task 


We  will  be  taking  a  trial  run  at  a  knowledge  elicitation  task  specified  as 
follows . 

The  knowledge  to  be  elicited  is  what  General  Whitehead  knows  which  he  would 
want  to  put  at  the  disposal  of  a  less  experienced  35A  (military  occupational 
specialty),  who  is  preparing  to  take  up  the  post  of  divisional  G-2  in  the  Ft. 
Leavenworth  "time  stress"  scenario  at  the  time  in  question. 

Knowledge  is  limited  to  that  which  is  not  available  in  existing  publications, 
nor  routinely  learned  at  a  junior  school  such  as  at  Ft.  Leavenworth.  It  may 
include:  knowledge  which  is  situation  specific  (i.e.,  as  regards  geography, 
threat,  mission,  etc.);  is  too  new  or  controversial  to  be  embodied  in  existing 
documents;  organizationally  sensitive  (such  as  using  discretion  in  "jumping 
channels");  goes  beyond  general  doctrine  (e.g.,  in  handling  exceptions  or  gen¬ 
erating  conditional  decision  rules). 

To  limit  the  scope  of  the  knowledge  further,  we  will  focus  on  certain 
separable  issues,  within  the  more  general  area  of  intelligence  collection 
management,  and  will  also  limit  ourselves  to  action  guidance  (as  contrasted 
with  description  or  causal  generalization) . 

We  may  need  to  fine  tune,  on  the  spot,  a  still  finer  definition  of  knowledge 
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One  is  to  use  the  same  task  and  subject  but  try  out,  more  deliberately, 
specific  techniques  and  intellectual  structures.  Another  would  be  to  repeat 
to  try  a  slightly  different  task  on  basically  the  same  subject  and  subject 
matter,  for  example  changing  the  assumption  about  who  is  to  be  the  receiver  of 
the  elicited  information  and  how  (e.g.,  a  computerized  expert  system)  or  pur¬ 
pose  (e.g.,  the  evaluation  of  training  rather  than  training,  itself  or  com¬ 
munication  between  expert).  Alternatively,  we  could  try  a  different  type  of 
subject  matter  with  the  same  or  different  subject  (e.g.,  descriptive  or  causal 
knowledge  rather  than  action  guidance,  which  I  think  this  one  will  be). 

In  any  case  we  will  want  to  step  back  and  think  what  if  anything  we  have 
learned  from  this  small  data  point  which  might  influence  the  general  conduct 
of  the  project. 
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ATTACHMENT  A-4:  INITIAL  STATEMENT  OF  KNOWLEDGE 

Memo  For  New  Division  G-2 

From  Ennis  Whitehead 

Date:  November  27,  1 986 

Subject:  Some  Suggestions  for  Your  New  Undertaking 

1.  Use  the  basics  from  your  training  and  previous  experience 

o  knowledge  of  intelligence  process,  organization,  system,  SOPs, 
forms,  etc. 

o  Knowledge  of  enemy:  organization, doctrine,  weapons,  capabilities, 
perceived  strengths  and  weaknesses 

o  knowledge  of  collection  resources  and  capabilities:  within  division, 
higher  echelon,  adjacent  (US  and  foreign),  compiementary  resources 
controlled  by  others  (G3,  Eng,  FA,  etc) 
o  knowledge  of  counterintelligence 

o  understand  how  to  make  the  system  work  (formal  and  informal), 
o  understand  how  to  use  people  (this  is  not  a  one  man  expert  system) 
o  develop  and  train  subordinates 

2.  Adapt  the  basics  to  the  wartime  situation. 

o  relate  prior  knowledge  to  current  situation 
o  anticipate  and  interpret  commander's  needs  and  guidance;  think 
ahead  24  to  48  hours  or  longer(not  done  in  this  example), 
o  focus  on  the  important;  prioritze  the  intelligence  effort 
o  identify  and  discard  erroneous  perceptions  and  knowledge;  adapt 
quickly  to  reality  of  situation 

o  use  all  collection  means  to  gain  information  (example:  use 
Bundespost  to  call  behind  enemy  to  learn  what  is  happening; 
interview  German  border  guards  and  local  officials  escaping  from  captured 
areas). 

o  meet  and  assess  key  players  in  the  division  intell  operation  (own 
staff,  Bde  S2s,  Artillery,  Aviation,  corps  G2  staff,  adjacent  intell  staffs); 
get  rid  of  incompetents  quickly;  find  the  quality  people. 

o  establish  a  good  informal  comnrso  system;  direct  contact  is  needed 
to  put  flesh  on  bones  of  periodic  intel!  reports,  etc 

o  give  attention  to  detail;  this  is  a  detail  business  with  an  over 
abundance  of  fragments  of  information;  create  a  system  which  provides 
the  mechanism  for  handling  detail;  check  it  periodically  to  see  if  it  is 
working,  but  don't  get  lost  in  the  detail  yourself. 
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o  seek  advice;  keep  an  open  mind;  challenge  your  perceptions  and 
assumptions;  challenge  the  assumptions  and  perceptions  of 
others  (commander,  63,  etc). 

o  remember  that  almost  all  incoming  reports  are  estimates  or 
judgemental  deductions  based  on  limited  information  and  inexperienced 
(first  days  of  war)  personnel,  often  transmitted  in  haste,  under  pressure. 

o  Verify  peacetime  perceptions  of  enemy  tactics,  weapons, 
capabilities,  concerns;  for  example,  are  E.  German  or  Polish  forces  being 
used?  which  of  our  weapon  systems  are  most  effective;  how  to  does  the 
enemy  find  its  counterbattery  targets  (Sigint,  radar,  airborne  artillery 
observers,  etc). 

o  Make  sure  the  flash  reporting  system  works  so  that  critical 
information  is  gotten  to  the  commander  and  G3  quickly. 

o  Timeliness  is  more  critical  that  total  accuracy;  G2's  judgement  can 
compensate  for  accuracy,  but  not  for  timliness. 

3.  Comments  on  Current  Intel  Summary 

o  Inability  of  MRDs  to  continue  attack  is  based  on  supply  problem 
while  them  combat  capability  is  relatively  unimpaired;  how  do  we  know 
the  supply  problem  is  real?  Remember  there  is  a  one  to  four  hours  lag  in 
information  from  the  frontline  units.  I  would  have  estimated  that  the  two 
MRDs  would  continue  their  attacks  and  the  9TD  could  be  attacking  through 
one  of  them  right  now(0600).  Otherwise  why  would  that  division  come 
within  artillery  range,  unless  it  was  about  to  pass  through  and  continue 
the  attack.  Also  the  second  echelon  7TA  is  so  close  that  it  could  also  be 
moving  forward  to  attack  after  topping  off  its  vehicles  (a  temporary  stop 
outside  artillery  range  to  make  final  preparations). 

o  The  above  comments  lead  me  to  set  these  high  priority  collection 
requirements  for  everyone  who  has  a  capability  to  provide  some  indication: 

-Is  the  9TD  moving;  are  there  any  indications  that  it  will  attack 
within  the  next  2,  4,  8,  12  or  24  hours?  (Commander  may  be  influenced  in 
his  course  of  action  if  he  knew  the  answer  to  that  question). 

-Report  any  indications  of  attack  preparations  or  forward  movement 
of  7TA;  Are  there  any  indications  where  and  when  it  will  attack  in  next 

4,  8,  12.  24  hours?  Indicators  may  include - . 

-Report  any  information  or  indicators  that  nuclear  weapons  may  be 

used 

-Watch  the  north  flank  and  the  boundary  with  the  23d  Panzer 
Division.  Corps  and  national  boundaries  are  desirable  attack  points;  send 
an  experienced  officer  to  the  German  CP  to  get  a  good  update;  establish  a 
special  intell  liaison  team  with  commo  if  it  does  not  already  exist. 

A-74 


ATTACHMENT  A- 5:  SUMMARY  (KBL) 

ROUGH  NOTES  ON  KNOWLEDGE  ELICITATION  SESSION 
24  NOVEMBER  1986 
(KBL) 

\ 

[The  following  are  some  rough  notes  on  Monday's  knowledge  elicitation  session 
with  General  Whitehead.  I've  mostly  just  tried  to  chronicle  the  session  as  it 
happened.  Although  some  long  dialogues  just  have  the  main  points  summarized, 
the  notes  basically  follow  the  flow  of  the  session.  Contributors  are  iden¬ 
tified  by  their  initials.  Anything  inside  square  brackets  is  my  editorial 
comment.  My  summary  follows  the  discussion.) 

RVB  began  by  setting  context.  Rapid  pass  at  sample  elicitation.  Context  is 
training  of  new  G2  fresh  out  of  school.  EW  to  take  role  of  seasoned  G2  trying 
to  impart  his  wisdom.  Specifically,  to  concentrate  on  getting  across  info 
that  wouldn't  be  learned  in  school.  Problem  focus  is  information  collection 
management . 

The  schedule  was: 

1.  General  flailing  about,  getting  feel  for  problem. 

2.  Focus  on  a  specific  problem:  method  for  tasking  specific  sub-units  & 
intel  assets  to  locate  enemy  command  posts. 

3.  A  more  directed  elicitation,  to  be  directed  &  focused  by  RVB. 

4.  Open  to  audience  to  try  out  other  methods. 

1 .  General.  Feel  for  problem. 

RVB  began  by  asking  for  any  general  nuggets  of  wisdom,  before  moving  into 
Fulda  Gap  scenario.  EW  responded:  fl)  G2 ' s  problem  is  to  adapt  what's  learned 
in  training  to  real  life  wartime  situation:  (2)  more  than  just  knowledge  is 

involved- -must  understand  how  to  use  people, 

✓ 

EW  then  jumped  into  scenario.  Wouldn't  trust  previous  G2 ’ s  assessment  because 
of  conflict  betw.  reports  (9th  TD  stationary;  7TA  not  attacking)  &  Red 
doctrine  (don't  sit  within  artillery  range;  7TA  is  50  km  out,  but  doctrine 
says  reserve  sits  at  100  km).  Also,  doctrine  &  history  say  it's  good  to  at¬ 
tack  on  boundary,  especially  between  nat'l  forces.  Takes  report  4  hrs  to  get 
to  us,  and  they  may  have  started  moving  in  that  time.  Would  send  someone  up 
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to  23rd  Panzer  to  eyeball  situation. 


General  tenets: 

o  Think  further  ahead  than  the  current  battle.  24-48  hours  out. 

o  Learn  to  use  all  intel  resources,  both  under  your  control  and  not. 

o  Don't  accept  assumptions  of  intel  report.  All  info  is  based  on  es¬ 
timates.  Challenge.  Try  to  confirm. 

o  Speed  (get  info  to  cmdr)  more  impt  than  accuracy,  but  give  judgment  (if 
you're  not  sure,  give  indication  of  uncertainty?]. 

[Note:  EW  seemed  to  be  more  able  to  articulate  these  general  tenets  in  the 

context  of  the  example.  RVB's  instructions  were  to  give  general  wisdom  before 
example;  EW  jumped  to  example,  then  generalized.  This  happened  repeatedly 
during  the  session.] 

2 .  Tasking  subunits:  CP  location 

EW  said  this  was  not  the  G2's  problem.  He  is  more  interested  in  what  enemy 
units  are  doing.  Where  CP  is  located  is  secondary.  This  sub-problem  was 
basically  dropped  from  that  point  on. 

2a.  Collecting  information  on  enemy  movement 

EW  clearly  thought  the  most  important  problem  was  finding  out  whether  9TD  and 
7TA  were  moving  in  to  attack,  and  where.  He  started  listing  resources  that 
could  be  used  for  finding  this  out.  Use  Corps  assets  (Mohawks,  communications 
intercept);  ask  division  ELINT  people  to  look  for  radar  indicative  of  move¬ 
ment  . 

There  ensued  some  questions  relating  to  how  he  decided  which  information  col¬ 
lection  was  a  priority.  RVB:  how  would  he  transmit  this  to  G2  so  he  could 
replicate? 

EW:  Major  consideration:  what  is  most  serious  threat  to  division?  [He  kept 

coming  back  to  this  point- -his  overall  strategy  seemed  to  be  to  try  to  collect 
more  info  on  most  serious  potential  threat.]  We  can  handle  guys  attacking 
now,  but  not  9TD  (1  hr  away)  or  big  force  in  rear  (4-6  hrs  away  &  5x  our 
strength).  The  earlier  I  can  give  hard  info  to  cmdr,  the  earlier  he  can  move 


to  meet  It.  He  can  cell  higher  HQ  &  geC  Chelr  reserve,  buc  Cakes  hours  Co  gee 
Chen  here,  so  need  info  before  need  is  criCical. 

CC:  How  do  you  know  which  info  co  quescion  &  which  Co  accepc?  Are  certain 
sources  assumed  to  be  "hard  data"  and  others  assumptions ;  you  believe  hard 
data  &  not  assumptions? 

EW:  Used  judgment  to  assume  9TD  had  been  located.  [Later  said  map  would  have 
?'s  unless  data  were  very  good;  this  ID  was  unusually  precise  for  military 
map.  Indicative  of  very  good  information.]  Hedged,  maybe  that  was  hasty  as¬ 
sumption.  Again  noted  highest  collection  priority  should  be  greatest  threat 
to  div. 

Next  were  some  comments  on  methodology:  RVB:  candidate  -  recreate  scenario 
in  presence  of  expert.  EW:  need  knowledge  expert  to  pick  up  anything 
elicitors  have  missed. 

Dialogue  between  CC  &  EW:  need  to  narrow  possible  Red  capabilities  into  few 
manageable  CA's.  EW  listed  a  few  (continue  attack;  commit  9TD;  move  up  7TA) . 
Always  worried  about  unknown  forces.  Nuclear,  threat  in  rear.  Didn't  answer 
Q  about  whether  that  means  you  shouldn’t  commit  everything. 

RVB:  does  EW  just  have  heuristics  about  what  kind  of  knowledge  is  needed,  or 

does  it  derive  from  loss  associated  with  doing  it  wrong?  Is  knowledge  or¬ 
ganized  around  valuable  items  of  info  or  actions  predicated  on  info  (esp.  how 
actions  &  therefore  consequences  would  change  if  we  had  the  info)?  EW 
reiterated:  worry  about  greatest  threat  to  accomplishment  of  mission;  time  is 
crucial.  [This  seems  to  imply  loss -driven- -  except  what  if  there's  nothing  we 
can  do  if  greatest  threat  materializes?  Unfortunately,  we  didn't  ask  him 
that . ] 

Dialogue  in  which  TM  asked  how  much  of  task  was  role  playing.  EW:  much  of  it. 
Need  to  be  sensitive  to  how  war  (now)  is  different  from  peacetime  exercises. 
Again  noted  conflict  betw.  doctrine  &  report  that  9TD  not  moving. 

CC:  is  reporting  cmdr  specific?  EW:  cmdrs  are  different.  Some  get  into  G2 
task,  have  "sense"  of  battlefield.  [Never  really  said  whether  he'd  report 
differently  to  the  2  types.] 
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RVB:  what  will  G2  do  better  if  he  absorbs  your  lessons?  EV:  Hard  question, 
o  Prioritize.  Soot  check  detailed  info,  but  concentrate  on  hi  priority 
things . 

o  Take  full  advantage  of  resources. 

Doesn't  know  whether  there  are  gaps  in  knowledge  of  G2  fresh  out  of  Lev.  In¬ 
tuition  is  yes.  TM:  you  concentrated  on  sources  of  uncertainty.  Any  you 
might  not  anticipate  from  training?  EV:  again  commented  on  conflict  betw. 
doctrine  &  intel  report  9TD  stationary. 

MSC:  You  observe  a  conflict  betw  doctrine  &  report,  and  want  to  find  out  more 
info.  What  do  you  do  if  you  can't  get  more  info?  EV  responded  by  listing 
other  sources  of  information.  MSC  said  assume  all  sources  are  down.  EW  said 
if  there  were  movement  you'd  be  able  to  find  out.  If  no  evidence  of  movement, 
he'd  tell  cmdr  they  weren't  moving.  Would  include  a  measure  of  his  uncer¬ 
tainty. 

3 .  Structured  approach 

Proceed  to  a  step  through  simulation.  EW  was  asked  to  "play"  out  the 
scenario,  both  making  decisions  &  imagining  events  unfolding.  [If  we  were  to 
do  this  for  real,  we'd  want  different  people  playing  different  parts.  Perhaps 
an  expert,  a  greenhorn  G2,  and  someone  to  generate  events- -all  different 
people.]  Context:  4  hrs  have  elapsed  &  cmdr  has  decided  on  CA3  (the  balanced 
one).  Time  is  1000  hrs  &  we're  now  implementing  CA3  [implicit  assumption  that 
no  new  information,  esp.  about  movement  of  9TD  or  7TA,  has  occurred], 

EW  says  cmdr  will  withdraw  some  of  BDE  forces  to  build  reserve;  that  will  be 
done  now,  in  daylight  (assuming  they  can  be  reasonably  safe  from  air  attacks). 

EW:  too  late  for  G2  to  contribute  much  towards  what's  happening  right  now. 

He  should  have  antennae  out,  looking  24  hrs  ahead.  Sensitize  analysts  to  need 
to  give  info  quickly. 

Asked  to  generate  continuation  of  events,  EW  gives  several  possibilities  (Red 
continues  attack;  Red  moves  9TD  up.  If  second,  we  should  get  indications  -  - 
radio  traffic  &  other  signatures- -  there  are  1000  vehicles.) 
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As suae  9TA  attacks  @  1000  hrs,  in  center.  Boundary  of  the  2  BDEs .  Passing 
thru  1st  echelon  could  take  1-20  hrs.  Key  problem  is  finding  out  where  is 
main  attack- -you  would  soon  know  where  you  were  taking  the  heaviest  attack. 

RVB:  what  about  them  fooling  us  with  a  small  attack  @  wrong  place  to  get  us 
to  commit?  EW:  you  would  expect  that  if  attack  lines  had  stabilized. 

They've  been  steadily  moving  forward,  so  it's  less  likely.  In  general,  all 
the  Red  forces  have  order  to  carry  out  secondary  attack,  but  attack  will  be 
heavier  @  location  of  primary  attack.  Purpose  of  secondary  attack: 
o  confuse  you 
o  hold  your  forces  in  place. 

You  will  know  soon  where  main  attack  is. 

Assume  only  penetration  is  one  area.  G2’s  jbb:  make  sure  you're  not  being 
fooled.  Don't  want  premature  commitment  of  reserve.  Goal:  find  out  who's  in 
main  attack.  Info  sources:  radio  (at  least,  you  can  find  out  #  of  nets  up), 
prisoners.  Ask  your  forces  to  continue  concentrating  on  9th.  Most  of  your 
collection  resources  are  with  BDEs.  If  lost,  try  to  replace  them. 

RVB:  what  might  bad  G2  do?  EW:  [didn't  really  answer  question]  Need  to  adapt 
system.  RVB:  example?  EW:  ask  for: 

o  AVN  BDE's  Scout  helicopters  (tell  you  size  of  penetration) 
o  Artillery  (to  locate  their  artillery.  Signals/trajectory) 
o  Additional  surveillance  resources  from  Corps 

o  Might  want  G-3  to  establish  recon  &  surv.  mission.  Find  out  where  enemy 
might  be  going.  Use  air  cavalry  SQN  of  reserve.  (G3  might  not  go  along 
if  low  on  helicopters,  n  o*  them  to  fight.  Can't  have  too  many  flying 
hrs . ) 

General  situation:  attack  heavier  than  previously.  Led  us  to  believe  9TD  at¬ 
tacking.  Division  is  responding.  EW  wants  to  know  what  9TD  is  doing.  Also 
watching  7TA. 

What  if  later,  9TD  attacks  on  north  flank?  Earlier  attack  was  secondary. 

Then  you  would  have  to  shift  resources.  Adjust  your  own  easily  adjustable 
collection  resources. 

o  CO  of  surv  radars  (1  PLT  each  BDE)--not  easily  movable, 

o  3  helicopters  to  collect  ELINT  (radar)  signals.  Put  them  up. 
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o  Very  quickly,  move  ground  based  collection  centers  up  north, 
o  Ask  for  help  from  Corps . 

Things  you  would  have  done  earlier  to  hedge  against  this  possibility: 
o  You  would  have  been  talking  to  Corps  all  along.  Maybe  he  would  have  had 
tipoff  earlier  &  let  you  know. 

o  You  would  have  gone  up  to  talk  with  Germans.  (Certainly  use  secure  phone 
if  you  had  it.)  You  would  take  along  someone  trusted  who  knew  German, 
o  Would  have  kept  some  helicopters  down  (they're  scarce  resource.  Limited 
flying  hours,  &  can  get  shot  down.  So  don’t  use  them  all  up  in  one 
place . ) 

Mid  afternoon  now. 

Generalities :  In  school  you  learn  enemy  capabilities  &  limitations.  There 

are  rules  of  thumb  [I  don't  know  whether  he  meant  you  learn  the  rules  of  thumb 
in  school  or  you  get  them  later  on  the  job].  G2  is  always  working  off  limited 
info  &  can  never  give  perfect  estimates.  Battle  of  bulge:  refused  to  inter¬ 
pret  what  they  saw  in  way  Germans  actually  were  deploying  their  forces.  [Was 
this  an  admonition  to  keep  an  open  mind?  Not  to  take  school  learning  about 
their  doctrine  too  literally?] 

Again,  a  discussion  about  what  EW  might  do/know  different  from  greenhorn. 
[Problem:  EW  doesn't  know,  and  we  have  no  baseline  for  comparison.]  JL  noted 
that  EW's  approach  was  very  different  from  S's  in  Leavenworth  experiment- -he 
focuses  on  general  strategic  issues,  they  on  micro  detail.  EW  noted  this 
could  be  because  they  were  G3's.  (But  shouldn't  the  G3's  be  the  ones  con¬ 
cerned  with  strategy?)  EW  has  learned:  (1)  Drive  system  110%  [this  is 
repeatedly  stressed- - I 'm  not  sure  of  an  operational  meaning];  (2)  Aircraft  are 
scarce  resource;  don't  use  frivolously  [is  this  a  qualification  to  (1)?]. 

CC  noted  that  much  of  EW's  knowledge  seemed  to  be  expressible  only  thru 
doing- -he  doesn't  have  access  to  it  (3  level  of  verbalization.  CW  expressed 
concern  that  verbalization  could  be  wrong- -e.g.  teacher  teaches  differently 
than  he  acts,  for  reasons  of  pedagogy. 
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4. 


JL  Cakes  over.  Asks:  You've  given  us  overall  structure:  Important  to  find 
out 

o  Where  Is  enemy? 
o  What  can  he  do? 

Why  does  this  matter? 

EW:  looking  for  threats  to  accomplishment  of  overall  mission.  Top-level 

goal:  find  out  info  about  threats  to  current  mission.  Secondary  goal: 

retain  ability  to  perform  next  mission. 

To  accomplish  this,  (1)  describe  to  cmdr:  what  are  greatest  threats  impacting 
on  his  mission;  (2)  tell  him  what  enemy's  going  to  do  near  term  &  long  term 
(long  term  —  24-48  hrs  in  Europe;  months  in  Vietnam). 

To  find  out  what  threats  are,  you  look  at  forces,  terrain,  weather.  [METT-T] 
If  Red  had  AF,  would  look  at  it  too.  Look  @  forces  in  terms  of  force  ratios. 
[Note  that  this  is  only  second  time  he's  explicitly  mentioned  force  ratios. 
Recall  earlier  he  said  7TA  was  5x  our  strength.)  Force  ratios  usually  just 
match  div  for  div,  but  should  have  mobility,  capability  (firepower),  previous 
practices  (agression)  factored  in  somewhere.  G2  does  enemy,  G3  does  friendly 
[I  assume  with  help  of  Gl],  force  ratio  is  computed  jointly.  Look  @  long  term 
threats:  size,  distance,  capability  to  initiate  attack. 

JL:  how  to  decide  on  capability  to  attack?  EW:  Force  ratio:  doctrine  says 
we  can  defend  3:1.  He'd  like  6:1  to  attack.  Speculates  about  utility  of 
designing  workstation  to  improve  our  ability  to  do  these  computations. 

JL:  How  to  determine  distance  constituting  a  threat?  EW:  Road  network,  how 
long  to  travel  [this  would  factor  in  both  speed  &  equipment],  doctrine.  Road 
network:  does  it  permit  diversion?  Distance,  location,  terrain:  can  they 

get  to  you?  Standard  [doctrinal]  rates  of  march- -you  can  compute  time  to 
move.  Taught  in  book.  Doctrine  says  division  worries  24-48  hrs  ahead. 

JL:  Capability  to  initiate  attack?  [He  asked  this  before.] 


EW;  Force  ratio.  [Same  answer  as  before].  Plus  what  they've  been  doing, 
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e.g.  attacking  plus  reserve  moving  up.  Reserve  plus  guys  on  line. 

JL:  Look  at  map  &  tell  me  what  are  greatest  threats  to  you. 

EW:  Worst  case,  they  have  fuel  &  orders;  ready  to  move.  Calculate  time  to 
1st  place  they  could  attack,  looking  @  road  net.  80  km  @  20  km/hr.  Might  be 
@  pt  of  contact  In  4  hrs.  If  they  need  to  break  into  contact  formation  &  fuel 
up  it  would  be  8-12  hrs.  Might  take  advantage  [of  what?],  attack  1st  light 
tomorrow  AM.  Or  could  come  late  PM. 

AM  gives  you: 
o  more  time  for  recon 

o  easier  passage  of  guard  for  reserves  who've  never  seen  this  battlefield 
o  need  more  time;  could  have  air  support,  artillery  support 
Better  surprise  at  night.  Doctrine  supports:  better  prep  AM;  better  surprise 
PM  [this  generalization  was  prompted  by  JL]  . 

JL:  tell  us  more  about  factors  contributing  to  success  of  attack.  EW  started 
wargaming  as  if  he  were  enemy  cmdr:  if  I  can  move  div  column  40  km  at  night 
(if  I  don't  get  stuck)  then  maneuver  means  everything.  If  I  can't  make 
breakthrough  of  crust,  then  I  want  to  line  up  everything.  Options:  (1)  hasty 
night  attack;  (2)  move  troops  to  forw  assembly  area;  recon;  attack  1st  AM 
light. 

There  followed  a  discussion  of  how  EW  had  taken  a  high-level  question  (factors 
contributing  to  success  of  attack)  &  responded  with  low-level  generation  of 
for- instance .  It  was  noted  that  EW  was  less  comfortable  just  stating  general 
principles;  more  comfortable  illustrating  them  first  by  example,  then 
(sometimes  with  prompting  from  elicitors)  abstracting  general  principles.  EW: 
you  want  general  principles?  9  Principles  of  War.  Then  admitted  he  couldn't 
even  list  them  all;  certainly  does  not  organize  knowledge  by  them. 

EW  then  abstracted:  criterion  of  successful  attack:  Be  on  objective.  [This 
does  not  to  me  seem  to  be  a  natural  generalization  of  the  above  example,  ex¬ 
cept  at  some  remove:  Overall  goal  -  be  on  objective.  Means  to  that  goal  are 
surprise  &  being  well  prepared.  PM  attack  supports  1st;  AM  attack  supports 
2nd.  So  you  have  to  think  about  how  Red  will  weigh  &  trade  off.] 


[The  following  is  an  attempt  to  summarize  some  of  the  main  points  of  the  dis¬ 
cussion. 

Overall  objectives  of  G2  (order  of  priority): 

1.  Identify  greatest  potential  threats  to  mission  accomplishment. 

2.  Retain  ability  to  perform  next  mission. 

To  do  this:  Remember,  all  info  is  uncertain. 

Sub -goals : 

a.  Prioritize  greatest  info  needs. 

b.  Try  to  collect  info  about  hi  priority  items. 

c.  Give  commander  prompt  (even  if  incomplete)  info  while  doing  b. 

To  do  a: 

i.  Most  important  problem:  where  is  he  &  what  is  he  doing? 

ii.  Where  do  you  suspect  your  info  is  wrong  or  incomplete? 

o  Where  Red  doctrine  conflicts  with  your  intel  estimates 
o  You  know  something  about  which  sources  are  usually  reliable 
o  There  is  a  standard  G2  language  used  on  maps  that  tells  you  how  good 
your  info  is.  (i.e.  ?  if  ID/location  is  uncertain) 

iii.  Which  area  of  uncertainty/incompleteness  has  most  serious  consequences? 
o  Enemy  capability 

Size 

-  Distance  (converts  to  time) 
x  Road  network 

x  Capability  of  Red  units  to  move 
x  Doctrine 

Capability  to  attack, 
x  Force  ratio 
x  Practices  (agression) 
x  Capabilities  (firepower) 
o  Factors  influencing  successful  attack 
Surprise 
Preparation 
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x  Recon 

x  Support:  Air  &  Artillery 
Objective 


iv.  Look  24-48  hrs  into  future. 

v.  Keep  an  open  mind  &  be  ready  to  act  if  events  disconfirm  your  hypotheses 
To  do  b: 

i.  Use  all  your  resources  110% 

o  Assets  under  your  direct  control 
o  Corps  assets 

x  Keep  in  contact  with  Corps 
x  Give  them  specific  requests 
o  International:  keep  in  touch  with  Germans 

ii.  Take  account  of  limitations,  scarce  resources. 

o  Flying  hrs  limited,  can  lose  helicopters/planes, 
o  Don't  over commit;  hedge  against  unexpected. 

x  Keep  sou.e  reserve,  esp.  on  forces  that  can’t  be  moved  quickly. 

iii. Spot  check  details,  but  don't  get  enmeshed  in  them. 

To  do  c: 

i.  Give  cmdr  your  best  current  estimate. 

ii.  Give  him  your  best  judgment 
o  Where  you  are  uncertain 

o  What  consequences  could  be  if  you're  wrong 


Notes  on  elicitation  methodology: 

o  There  seemed  to  be  some  concern  that  it  was  too  unfocused,  too  general. 
That  we  weren't  getting  anything  useful.  Others  thought  that  we  were 
getting  a  lot. 
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It  is  pretty  clear  that  General  Whitehead's  knowledge  is  more  accessible 
at  the  detail  level  than  at  the  level  of  stating  general  principles  out 
of  context.  If  this  generalizes,  it  has  clear  implications  for  the 
methodology  we  use. 

o  It  was  agreed  that  a  good  candidate  method  would  be  to  observe  a  map  ex¬ 
ercise  and  be  allowed  to  stop  and  ask  questions  at  any  point.  Observing 
the  map  exercise  at  Leavenworth  will  not  be  good  enough  because  we  can’t 
ask  questions,  but  it  will  give  us  useful  information  that  will  help  us 
to  ask  good  questions  the  next  time. 

o  It  was  also  generally  agreed  that  we  need  a  structural  template  to  fit 

what  we're  getting  into.  (Of  course,  you  need  sessions  like  this  one  to 
help  you  develop  your  structure.)  John  says  he’s  well  on  the  way  to 
developing  a  structure.  I  think  soon  we  should  have  a  meeting  at  which 
he  presents  his  preliminary  ideas  for  group  feedback. 
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ATTACHMENT  A-6:  SUBJECT'S  RESPONSE 


TO: 

Rex  Brown,  DSC 

FROM: 

Ennis  Whitehead,  BAL 

DATE: 

November  26,  1986 

SUBJECT: 

Comments  on  Trial  Session 

I  thought  the  session  was  very  useful.  It  showed  that  a 
simple  map  exercise  or  case  study  can  provide  the  medium  for  trying 
out  several  elicitation  methodologies.  In  fact,  it  may  become  our 
principal  model,  because  it  is  so  well  understood  by  military 
officers.  On  the  other  hand,  we  also  need  to  use  other  approaches 
to  broaden  the  total  project. 

Three  comments  on  the  session. 

•  What  knowledge  are  we  trying  to  elicit?  Facts? 
Conceptualizations?  Estimating  rules?  Procedures?  Integrating 
methodologies?  We  need  to  define  "knowledge"  better. 

•  How  many  data  points  do  we  need?  I  believe  BAL  and  the 
Army  school  system  are  the  best  sour c&6, 

•  Where  are  we  headed?  We  need  to  find  a  "purpose"  for 
developing  elicitation  methodologies.  We  cannot  afford  to  avoid 
that  issue  very  long. 

I  thought  John  Leddo  was  very  well  prepared  with  his  line  of 
questioning.  I  assume  that  all  "elicitations"  will  be  equally  well 
prepared  when  we  go  into  data  collection. 
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ATTACHMENT  A-7:  MODERATOR'S  PERSPECTIVE 


MEMORANDUM 


TO: 

Knowledge  Elicitation  Project  Team 

FROM: 

P,\  O' 

Rex  V.  Brown  <  > 

SUBJECT: 

Trialcase  Exercise 

DATE: 

December  5,  1986 

REFLECTIONS  ON  SESSION  WITH  GENERAL  WHITEHEAD 

The  following  are  thoughts  on  some  of  the  issues  I  had  hoped  the  trial  elicitation  session  would 
cast  light  on,  for  project  planning  purposes.  Due  to  the  very  tentative  and  exploratory  nature  of 
the  session  and  the  fact  that  only  one  subject  and  one  possible  elicitation  task  was  addressed,  in¬ 
clusions  can  only  be  suggestive  and  impressionistic. 

Type  of  Elicitation  Task(s)  We  Should  be  Developing  Methods  For 

We  were  testing  a  very  limited  kind  of  elicitation  task:  the  knowledge  an  expert  G-2  would  want 
a  trained  but  inexperienced  G-2  to  know,  on  taking  up  a  divisional  G-2  assignment,  in  the  early 
stages  of  hostilities  with  the  Soviets  in  West  Germany.  If  the  case  we  took  is  at  all  representa.ive, 
while  it  is  clear  that  an  expert  would  have  a  great  deal  to  contribute  to  the  trainee  (beyond  what 
the  trainee  would  have  picked  up  in  regular  school),  it  is  not  obvious  that  the  best  way  to  do  it  ir 
along  the  lines  we  were  exploring.  I.e.,  extracting  the  knowledge,  and  recording  it  somehow  (e.i 
in  text,  video  or  computer),  with  a  view  to  getting  it  into  the  trainee  later.  The  alternative,  of 

having  the  expert  interact  with  the  trainee  directly  (e.g.,  in  a  pre-posting  briefing',  might  be  out- 

/ 

side  our  scope. 

I  am  not  suggesting  we  give  up  on  the  training  function.  It  does  seem  clear  to  me,  however,  that 
the  selection  of  knowledge  to  be  elicited  is  highly  dependent  on  the  use  to  which  it  is  to  be  put 
(and  so  is  the  method  for  eliciting  it).  If  the  purpose  is  training  (as  here)  knowledge  elicitation 
probably  needs  to  be  developed  in  the  context  of  a  broader  training  philosophy.  This  again, 
probably  takes  us  beyond  our  scope. 


Conversely,  if  we  are  planning  to  develop  some  kind  of  computerized  expert  system,  to  be  used 
as  a  decision  or  inference  aid  on  the  battlefield,  we  would  develop  it  in  a  way  that  has,  I  suspect. 
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little  overlap  with  the  training  function.  Knowledge  elicitation  for  trainee  evaluation  would 
probably  be  quite  different  again. 

In  any  case,  we  need  to  determine  what  product  or  products  we  will  eventually  try  to  deliver — 
i.e.,  methodologies  for  what?  There  is  probably  nothing  wrong  with  moving  out  in  more  than 
one  direction  initially,  with  a  view  to  possibly  settling  on  one  later.  However,  I  strongly  recom¬ 
mend  getting  feedback,  if  not  direction,  from  the  "client."  Discussion  with  some  key  people  at 
Huachuca  would  be  very  helpful. 

What  Kinds  of  Knowledge  do  we  want  to  be  able  to  Elicit 

The  underlying  thrust  of  the  knowledge  EW  was  giving  us  was  of  the  production  rule  type  e.g., 

"if  the  enemy  reserves  are  within  artillery  range  they  are  probably  on  the  move";  "other  things 
being  equal,  the  enemy  will  probably  attack  at  the  boundary  between  friendly  forces."  Much  of 
the  knowledge,  as  in  those  two  examples,  was  on  the  inferences  that  would  go  into  a  G-2’s  deci¬ 
sions,  rather  than  directly  as  decision  rules.  This  would  make  sense,  since  you  can  be  more  sure 
of  inferences  than  of  decisions  which  take  those  inferences  into  account,  and  also  take  into  ac¬ 
count  a  whole  lot  of  other  things  that  EW  would  not  know  about  (at  least  from  the  amount  of 
background  information  he  has  been  given). 

Much  of  his  knowledge  would  be  relevant  to  producing  enhanced  doctrinal  documents  (such  as 
field  manuals),  but  the  form  he  gave  the  knowledge  in,  was  almost  invariably  very  situation 
specific,  i.e.,  not  even  as  general  as  a  production  rule.  If  we  probe  into  why  he  made  these 
recommendations,  we  can  no  doubt  work  our  way  up  to  the  goals — means  hierarchy  (and  that 
might  be  a  useful  exercise  to  pursue). 

There  seem  to  be  two  distinctive  types  of  knowledge  we  uncovered  which  is  not  in  your  normal 
field  manual  material:  illustrative,  situation  specific  knowledge  which  implies  more  general 
principles;  and  considerations  to  take  into  account  when  coming  up  with  specific  inferences  or  ac¬ 
tions,  though  the  latter  presumably  would  be  in  the  manuals,  if  anyone  thought  they  were 
relevant. 

A  promising  body  of  knowledge  to  be  elicited  would  be  a  series  of  battle  sequences,  whose  func¬ 
tion  is  to  give  trainees  vicarious,  concentrated  battle  experience,  to  complement  what  they  get  out 
of  manuals.  The  enemy  and  non  G-2  friendly  components  would  be  made  plausible  (but  not 
otherwise  explained);  and  the  G-2  components  would  represent  the  expert’s  best  judgment,  (with 
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a  commentary  supplied  by  him.)  Alternatively,  if  available,  real  war  sequences  could  be  used, 
with  the  actual  (not  necessarily  best)  G-2  action.  Here  the  expert's  contribution  could  be  limited 
to  commentary  on  that  G-2  action.  Where  in  the  interpretative--generative  continuum  the  com¬ 
mentary  would  be  pitched  depends  not  only  on  how  the  expert  thinks,  but  how  the  trainee  thinks 
too. 

If  the  latter  type  of  knowledge  is  being  elicited,  it  might  be  put  in  the  form  of  training  materials 
e.g.,  texts  or  video — but  interaction  with  the  trainee  could  be  computer-aided. 

BealflgihR  .tte  St.eiL-ih,rc>ygh  AcacaaslUffi  Elicitation 

Some  variant  of  step-through  simulation  (itself  a  variant  of  wargaming)  looks  promising,  but  the 
tack  I  took  would  need  major  modifications.  Most  important,  in  particular,  the  team  generating 
the  non  G-2  sequence,  needs  to  be  distinct  from  the  expert  generating  the  G-2  responses  of 
primary  interest.  (For  example,  enemy  action  might  be  modeled  by  another  G-2  expert,  friendly 
action  by  G-3  etc.)  At  this  point  there  are  economy  and  deluxe  options  to  step- through. 

The  economy  model  would  be  static  pre-existing  (and  therefore  and  cheaply  available)  sequences, 
are)  be  presented  to  the  expert,  who  says  at  each  G-2  action  node  what  he  would  have  done  and 
why.  However,  the  sequence  would  then  continue  assuming  that  the  pre-existing  G-2  action 
written  into  the  sequence  was  the  one  adopted.  That  pre-specified  G-2  action  would  be  handled 
in  no  more  detail  than  any  other  item  in  the  sequence. 

The  deluxe  option,  the  real  step-through  version,  has  the  expert’s  G-2  input  integrated  into  the 
sequence.  I.e.,  the  "response  team"  and  the  expert  would  alternate  taking  into  account  everything 
that  has  gone  before.  In  this  way,  we  could  simulate  what  the  result  of  the  G-2  action  was,  with 
particular  attention  to  what  difference  it  made. 

What  Next? 

If  it  turns  out  that  the  client  wants  (and  is  to  be  given)  methods  for  eliciting  this  type  of 
knowledge,  then  I  suggest  we  try  to  put  the  deluxe  option  into  practice.  I  see  no  reason  not  to 
stick  with  the  Treysa  defense  scenario  since  we  are  relatively  up  to  speed  on  it.  For  this  we 
would  need  to  establish  a  "response  team",  probably  one  or  two  more  from  BAL’s  staff.  We 
would  allow  the  expert  to  ask  the  response  team  to  invent  realistically  any  information  they  had 
not  already  volunteered,  but  which  the  real  G-2  would  have  access  to  and  comment  on  why  he 
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did  what  he  did.  Ideally  the  response  team  would  specify  the  ‘bottom  line*  of  how  the  engage¬ 
ment  turned  out,  picking  out  anything  of  special  relevance  to  the  G-2  (for  example,  if  any  con¬ 
troversial  decisions  which  he  made  were,  or  were  not,  vindicated  by  the  outcome  of  the  battle). 
Our  view  of  knowledge  structures,  mental  models,  etc.  may  influence  (or  be  influenced  by)  the 
unfolding  of  this  experiment. 

ILnal  this.  What? 

Whether  we  pursue  this  tack,  depends  not  only  on  the  client’s  expressions  of  interest,  but  also  on 
ours.  I.e.,  if  we  don’t  go  after  this  type  of  product  what  will  we  go  after?  Probably  it  makes 
sense  to  test  the  waters  for  a  computer  based  expert  system,  without  delay,  if  that  is  the  most 
promising  alternative.  I  would  recommend  that  Craig  Cook  or  his  colleagues  try  to  scope  out 
such  an  exercise  (again,  perhaps,  using  a  trialcase  exercise)  to  the  point  where  we  can  visualize 
what  the  products  might  look  like  (and  for  what  purpose). 

We  can  seek  closure  after  the  Huachaca  meeting. 
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ATTACHMENT  A-8:  EXPERT  SYSTEM  PERSPECTIVE 

MEMORANDUM 


TO:  Rex  Brown 

FROM:  Craig  Cook 

SUBJECT:  Observations  on  Trial  Knowledge  Elicitation  Session  - 

11/24/86 

DATE:  December  8,  1986 


The  following  are  some  observations  and  ideas  about  what 
transpired  at  the  Subject  trial  session  and  how  it  might 
influence  the  direction  of  the  project.  The  purpose  of  this  memo 
is  to  log  these  various  points  for  future  discussion/ 
consideration . 

The  session  seemed  useful  in  opening  up  the  topic  area 
(Management  of  Tactical  Intelligence  Collection).  Several 
categories  of  knowledge  were  identified  -  primarily  by  example. 
Considerable  analysis  and  follow-up  sessions  presumably  would  be 
required  to  structure  the  knowledge,  complete  the  identif ication 
of  the  components  in  each  category,  and  build  the  appropriate 
associations  and  chains  of  reasoning  before  we  could  claim  to 
have  successfully  elicited  useful  knowledge. 

A  major  issue  in  any  elicitation  endeavor  would  seem  to  be 
"What  knowledge  is  being  sought?"  This  is  tied  to  the  question 
of  now  much  knowledge  the  novice  G2  can  be  reasonably  expected  to 
have  a  priori.  This  "basic  knowledge"  is,  itself,  a  fuzzy  set 
since  it  will  entail  not  only  formal  training  but  the  experience 
and  intuition  of  the  G2  and  therefore  vary  by  individual.  Hence, 
an  initial  give-and-take  question  and  answer  interchange  may  be 
necessary  in  any  session  so  that  the  expert  can  determine  what 
knowledge  (and  at  what  level  of  detail)  it  is  necessary  to 
impart. 

In  this  particular  session,  there  was  no  novice  G2  for  the 
expert  to  probe;  hence,  there  was  some  difficulty  for  the  expert 
in  deciding  how  much  to  say  in  a  response.  Further,  because  we 
as  elicitors,  did  not  have  a  broad  base  of  knowledge  in  the 
application,  we  are  unable  to  assess  the  quality  or  utility  of 
the  knowledge  provided. 

A  related  shortcoming  of  the  trial  session  involved  its 
incompleteness.  By  establishing  a  scenario  that  was  all 
hypothetical  we  asked  the  expert  to  make  many  assumptions: 
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.  Assume  there  is  a  novice  G2 

.  Assume  he  has  some  gaps  in  his  knowledge 

.  Guess  what  you  (the  expert)  think  those  gaps  would  be 

.  Assume  the  wartime  scenario  is  as  given 

.  Indicate  what  actions  you  would  take 

.  Assume  what  responses  you  would  get 

.  Indicate  what  intelligence  collection  resources  you  might 
have  available 

.  Guess  what  might  go  wrong 

.  With  all  tnis,  determine  what  the  enemy's  plans  are  (which 
is  unverifiable  since  the  scenario  is  incomplete). 

This  kind  of  open-ended  situation  would  seem  to  be  useful  for 
getting  the  expert  to  identify  many  potentially  relevant  pieces 
of  knowledge,  i.e.,  breadth.  However,  it  seems  inadequate  for 
eliciting  subtle  distinctions  that  contribute  to  the  expert's 
expertise  (over  that  of  a  novice  G2). 

In  the  session  as  it  transpired  the  elicitors  did  not 
generate  especially  sophisticated  questions  designed  to  elicit 
subtleties  in  the  expert's  knowledge.  One  result  of  this  may  be 
to  nave  given  the  expert  the  impression  that  the  elicitors  did 
not  know  enough  to  question  the  validity  of  what  the  expert  was 
saying.  Thus  there  may  not  have  been  sufficient  pressure  on  the 
expert  to  generate  high-quality  knowledge  (i.e.,  knowledge  which 
would  withstand  peer  review).  I  suspect  that  any  expert  would 
insist  on  the  right  to  review/revise  the  knowledge  prior  to 
submitting  it  for  peer  review. 

The  second  round,  John's  top-down  structured  interview 
approach,  seemed  to  elicit  specific  detail  (e.g.,  lists). 
However,  the  responses  to  direct  questions  also  seemed  to  be 
direct  answers  which  may  or  may  not  have  been  well  thought-out. 
There  appeared  to  be  some  motivation  for  responding  with  a  quick 
and  short  answer.  The  more  fundamental  questions,  i.e.,  those 
with  textbook  like  answers  seemed  to  be  slightly  irritating  to 
the  expert  (e.g.,  answers  like  "its  straight  out  of  the  book."). 
This  may  imply  the  need  to  do  adequate  preparation  so  that  basic 
knowledge  questions  are  avoided  or  at  least  asked  deliberately. 

The  expert  appeared  to  have  vast  quantities  of  knowledge 
about  the  domain,  but  it  did  not  seem  to  be  organized  for 
pedagogical  purposes  (i.e.,  structured  in  any  particular  way). 
The  expert  was  best  able  to  impart  his  knowledge  by  using 
examples  and  then  stating  some  general  principle  tnat  encompasses 
the  example  rather  than  presenting  a  theory  and  listing  a  set  of 
general  principles  followed  by  specific  examples.  This  may  be 
the  case  with  practitioners  (vs.  teachers)  and  may  need  to  be 
accommodated  in  the  methodology. 
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In  summary,  there  seems  to  be  no  clear  way  to  measure  the 
quality  of  the  elicitation  session.  Assessing  the  success  of  a 
session  depends  upon  the  assessor's  point  of  view  and  might 
involve  a  complex  trade-off  among  such  issues  as: 

.  Speed  of  the  session 
.  Volume  of  data/knowledge  obtained 
.  Richness  of  knowledge 

.  Novelty  of  knowledge  (over  textbook  "basic’'  knowledge) 

.  Rapport  between  elicitor  and  expert 
.  Utility  of  knowledge  extracted 
.  Breadth  of  knowledge  identified 
.  Depth  of  knowledge  obtained 
.  Validity  or  accuracy  of  knowledge  obtained 

Perhaps  it  is  not  appropriate  to  try  to  assess  the  absolute 
quality  of  a  single  knowledge  elicitation  session.  Since  any 
elicitation  endeavor  can  be  expected  to  involve  many  sessions 
(with  reviews  and  r epet i tion)  along  with  off-line  analysis, 
perhaps  it  is  better  to  assess  the  total  progress  (quality)  of 
the  elicitation  series  and  establish  specific  goals  for 
individual  sessions  (e.g.,  "This  session  will  open  up  as  many 
aspects  of  the  total  problem  as  possible."  "This  session  will 
concentrate  on  distinguising  the  factors  which  led  to  this 
particular  decision  or  choice."  etc.).  The  thing  we  may  want  to 
do  is  measure  the  quality  of  the  knowledge  base  developed  to 
date,  identify  gaps  in  it,  develop  a  plan  for  filling  those  gaps, 
and  iterate.  The  measure  of  quality/success  for  a  particular 
elicitation  session,  then,  deals  with  how  well  the  session  filled 
the  knowledge  gaps  (including  identifying  new  gaps). 
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ATTACHMENT  A-8  (continued) 


WORKING  PAPER 


TO: 

FROM; 

SUBJECT: 

DATE: 


Rex  Brown 
Craig  Cook 

Some  Thoughts  on  Knowledge  Elicitation 
December  8,  1986 


The  following  are  some  sketchy  ideas  that  might  be  worth 
developing  in  greater  detail. 


1.  KNOWLEDGE  ELICITATION 

Knowledge  acquisition  is  the  process  of  obtaining  knowledge 
from  a  variety  of  sources  and  representing  it  in  some  form  for 
automated  processing. 

Knowledge  elicitation  is  the  process  of  soliciting  and 
capturing  knowledge  from  a  human  source.  The  process  is 
proactive,  cognitive,  often  iterative,  and  one  which  seeks  to 
optimize  the  utility  of  the  knowledge  elicited. 

Knowledge  elicitation  is  a  two-way  communication  process. 
It  is  more  than  mere  rote  recording  of  the  knowledge  provided  by 
the  source.  It  entails  an  active  concept  formation  process  on 
the  part  of  the  elicitor  which  involves  data  and  relationships, 
procedures  and  techniques,  and  values  and  heuristics. 

A  key  component  of  knowledge  elicitation  is  the  formation  of 
questions,  the  answers  to  which  are  intended  to  help  clarify  the 
distinction  among  various  domain  concepts  and  the  reason  for 
various  decisions  or  choices  made  by  the  source.  In  this  sense, 
negative  information  (i.e.,  what  a  concept  excludes)  can  be  just 
as  valuable  as  positive  examples  in  refining  the  definition  of  a 
particular  concept. 

The  quality  of  a  knowledge  elicitation  endeavor  can  be 
measured  in  several  ways: 

•  Efficiency  -  The  speed  with  which  knowledge  is  transferred 
from  the  source  to  some  medium  by  the  elicitor  (including 
repetitions,  restarts,  and  refinements). 

.  Effectiveness  -  The  breadth  and  depth  of  detail  of  the 
knowledge  that  is  captured  (i.e.,  completeness,  precision, 
relevance)  as  compared  to  the  purpose  of  the  elicitation. 
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.  Usefulness  -  The  accuracy,  richness,  consistency,  or  utility 
of  the  knowledge  obtained  through  the  elicitation. 

The  quality  of  a  knowledge  elicitation  endeavor  may  be 
influenced  by  several  factors,  including: 

.  The  quality  of  the  knowledge  source  (depth  of  understanding, 
breadth  of  related  knowledge,  identification  of  critical 
issues,  clarity,  structuredness,  etc.) 

.  The  articulateness  of  the  knowledge  source  for  expressing 
the  knowledge  in  a  comprehensible  form 

.  The  quality  of  the  techniques  employed  by  the  elicitor  to 
draw  out  and  clarify  the  knowledge 

.  The  skill  of  the  elicitor  in  employing  those  techniques  and 
in  using  the  right  technique  at  the  right  time 

.  The  appropriateness  of  the  medium  used  to  convey  the 
knowledge  to  the  elicitor  (e.g.,  voice,  writing,  pictures/ 
diagrams,  performance  of  a  task) 

.  The  appropriateness  of  the  medium  used  by  the  elicitor  to 
capture  the  knowledge  for  further  analysis,  processing,  or 
training 

,  The  breadth,  depth,  and  sophistication  of  the  knowledge 
already  possessed  by  the  elicitor  (e.g.,  familiarity  with 
terminology,  background  concepts,  classic  examples) 

.  The  insightfulness  of  the  elicitor  and  his/her  ability  to 
rapidly  see  patterns  or  grasp  new  concepts 

.  The  discipline  of  the  elicitor  to  avoid  drawing  hasty 
conclusions,  making  unfounded  assumptions,  or  failing  to 
balance  breadth  and  depth  appropriately 


2.  KNOWLEDGE  TAXONOMY 

The  question  of  "what  is  knowledge"  is  a  major  philosophical 
area  probably  not  relevant  to  our  project.  However,  the  question 
of  "what  knowledge  are  we  eliciting"  seems  to  be  pertinent  to 
directing  the  elicitation  process.  Three  broad  schemes  for 
categorizing  knowledge  are: 

.  Epistomolog ical  -  By  subject  area 
.  Representational  -  By  how  it  is  represented 
.  Util i tar ian  -  By  its  use 


The  epistemological  approach  is  characterized  by  models  such  as 
academic  disciplines  at  a  university  or  the  Dewey  Decimal  System 
employed  by  library  science.  The  representational  approach  is 
characterized  by  such  aspects  as  mental  vs.  written,  textual  vs. 
pictorial,  structured  vs.  unstructured,  associative  vs.  logical. 

The  utilitarian  model  categorizes  knowledge  into  three  broad 
categories  according  to  how  the  knowlege  is  used  (not  what  it  is 
about  nor  how  it  is  represented  in  some  medium): 

(1)  Factual  Knowledge  -  Descriptive  knowledge  pertaining  to  real 
or  conceptual  objects.  For  example, 

.  Data 

.  Relationships 
.  Pictures/Diagrams 
.  Narrative  Text 

(2)  Procedural  Knowledge  -  Knowledge  whicn  is,  in  some  sense, 
executable.  For  example, 

.  Procedures 
.  Algorithms 
.  Skills 
.  Perception 

(3)  Judgmental  Knowledge  -  Evaluative  knowledge  which  influences 
choices,  decisions,  or  preferences.  For  example, 

.  Values 
.  Goals 
.  Objectives 
.  Constraints 

Factual  knowledge  (whether  true  or  false  or  unverified)  is 
what  the  source  knows.  Procedural  knowledge  is  what  the  source 
knows  how  to  do  (however  adept  or  inept).  Judgmental  knowledge 
deals  with  understanding  why  a  particular  piece  of  knowledge  is 
important . 

Knowledge  elicitation  involves  capturing  the  knowledge  in 
some  form  which  can  be  replicated  later.  This  knowledge  may  be 
stored  in  the  head  of  the  elicitor  or  it  may  be  written  down  in 
some  other  medium.  Knowledge  representation  deals  with  how 
knowledge  is  captured  in  some  medium.  Many  forms  of  knowledge 
representation  have  been  developed,  including: 

.  Textual  Descriptions 
.  Formal  Procedures 
.  Computer  Programs 
.  Mathematics  and  Logic 
.  Pictures,  Diagrams,  and  Charts 
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.  Rules  of  Behavior 

.  Frames  and  Data  Structures 

.  Heuristics  and  Guidelines 

.  Time  Sequenced  Scripts 

.  Neural  Networks 

.  Mental  Models 

The  appropriateness  of  a  knowledge  representation  depends  upon 

.  What  kind  of  knowledge  is  being  represented  (e.g.,  facts, 
procedures,  judgments) 

.  To  what  use  the  knowledge  will  be  put  (e.g.,  problem¬ 
solving,  task  performance,  strategic  planning,  assessment, 
training) 

.  The  intellectual  capabilites  and  background  of  the  knowledge 
interpreter,  i.e.,  the  entity  which  will  make  use  of  the 
knowledge  represented  (e.g.,  depth  of  detail,  level  of 
abstraction,  breadth  of  related  knowledge,  "common  sense,") 

The  selection  of  a  knowledge  representation  scheme  can 
influence  the  quality  of  a  knowledge  elicitation  endeavor  by 
facilitating  or  hindering  the  efficiency,  effectiveness,  and 
usefulness  of  the  elicitation  process.  But  it  is  not  clear  in 
wnat  ways  or  by  how  much  the  influence  could  be  felt.  Also, 
there  may  be  a  series  of  representation  schemes  that  can  be 
employed  and  the  knowledge  may  be  translated  from  one  to  another 
as  needed  (in  part  or  in  toto). 

Tools  could  be  developed  to  aid  the  elicitor  by 

* 

.  Quickly  capturing  the  knowledge  as  it  is  presented 

.  Saving  the  knowledge  for  subsequent  review  and  analysis 

.  Regenerating  the  knowledge  for  some  purpose  (e.g.,  training, 
explanation,  or  use) 

.  Assessing  the  consistency  of  the  knowledge  captured/ 
represented  and  identifying  inconsistencies 

.  Postulating  concepts  and  generating  experiments  or  questions 
to  refine  concept  definitions,  to  test  procedural 
instructions,  or  to  clarify  value  judgments. 

The  knowledge  possessed  or  employed  by  any  individual  in 
performing  a  cognitive  function  can  be  extraordinarily  large, 
diverse,  and  complex.  The  performance  of  a  substantive  task  may 
involve 
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.  The  retrieval  of  facts 

.  The  execution  of  procedures 

.  The  application  of  judgments 

all  occurring  concurrently  both  consciously  and  subconsciously. 
The  purpose  of  knowledge  elicitation  is  to  draw  out  this 
knowledge  in  some  systematic  way. 

A  knowledge  source  (e.g.,  a  human  expert)  can  easily 
demonstrate  his  or  her  knowledge  through  the  execution  of  a  task 
(solving  a  problem,  answering  a  question,  manipulating  a  device, 
recognizing  an  object,  generating  a  remark),  but  when  asked  to 
explain  (i.e.,  teach)  how  the  task  is  done,  the  source  may 
experience  some  difficulty.  Part  of  this  difficulty  may  stem 
from  the  communication  process  by  which  the  source  must  explain 
the  knowledge  to  the  elicitor.  This  process  is  generally 

.  Sequential  (vs.  a  potentially  parallel  mental  activity) 

.  Slow  (vs.  the  mental  speeds  with  which  the  source  uses  the 
knowledge ) 

.  Structured  (vs.  a  sometimes  associative  approach). 

The  process  of  converting  from  whatever  form  the  source’s  mental 
knowledge  may  take  to  a  form  that  is  communicable  and 
understandable  by  the  elicitor  may  impact  significantly  the 
quality  of  the  knowledge  elicitation  endeavor  (speed,  precision, 
completeness,  clarity,  etc.). 


3.  KNOWLEDGE  ELICITATION  METHODOLOGY 


The  purpose  of  this  project  is  to  develop  a  methodology ( ies) 
for  knowledge  elicitation.  Such  a  methodology  might  consist  of 

.  A  description  of  the  overall  knowledge  elicitation  process: 
its  goals,  its  stages,  its  cycles  -  and  the  purpose  and 
issues  associated  with  each 

.  The  identification  and  description  of  various  techniques  for 
eliciting  knowledge  from  a  human  source 

.  An  assessment  of  the  appropriateness  of  the  various 
techniques  for  eliciting  various  types  of  knowledge  (where 
"types"  may  have  several  dimensions) 

.  An  indication  of  the  importance  of  having  a  representation 
and  the  impact  it  may  have  on  the  quality  of  the  elicitation 
process 

.  An  analysis  of  various  of  the  knowledge  representation 
schemes  and  their  suitability  for  various  purposes  in  the 
knowledge  elicitation  process 


A- 100 


.  A  means  for  measuring  the  quality  of  the  knowledge  elicited 

and  of  the  knowledge  elicitation  process  itself. 

The  methodologies  should  have  a  measure  of  practical  utility, 
i.e.,  they  should  be 

.  Executable  and  understandable 
.  Applicable  to  several  situations 

.  Identifiable  as  to  when  they  are/are  not  applicable 
.  Measurable  as  to  whether  they  are  working  or  not. 

The  methodologies  should  be  couched  in  terminology/examples 
that  are  familiar  to  the  customer  (i.e.,  the  U.S.  Army)  but 
should  be  general  enough  to  apply  to  other  domains  as  well.  One 
criterion  that  may  effect  a  methodology  selection  may  be  the 
particular  Army  task  or  the  Army  knowledge  source. 

The  methodologies  should  result  in  a  capturing  of  knowledge 
that  is  suitable  for  a  variety  of  purposes: 

.  Training  future  Army  officers 
.  Evaluating  current  officers’  knowledge 

.  Developing  formal  procedures 

.  Coding  a  computerized  (expert)  system. 

A  major  difficulty  in  eliciting  knowledge  can  be  just 
getting  started.  If  one  envisions  the  source's  knowledge  as  a 
vast  network  of  highly  interconnected  nodes  with  no  beginning  or 

end,  then  there  may  be  no  "logical  place  to  start."  Probing  this 

network,  more  or  less  at  random,  can  provide  a  potentially  large 
number  of  paths  (associations  or  lines  of  reasoning)  in  many 
directions  leading  out  from  the  point  of  the  probe.  The  job  of 
the  elicitor  is  to  steer  the  session  in  such  a  way  as  to  identify 
the  various  paths  and  knowledge  chunks  so  they  can  be  captured 
(structured,  analyzed,  used,  etc.) 

The  source  is  concerned  that  his  or  her  utterances  are 
meaningful  to  the  elicitor  and  is  unsure  of  how  much  detail  (or 
generalization)  is  necessary  for  the  elicitor  to  understand  the 
knowledge  being  expressed.  There  may  be  a  tendancy  to 
oversimplify  (to  make  it  easier  for  the  elicitor  to  understand  or 
to  reduce  the  amount  of  work  the  source  must  go  through  to 
produce  the  knowledge).  For  example,  the  source  may  espouse 
"textbook  rules"  and  not  go  into  the  details  of  how  exceptions  to 
these  rules  should  be  identified  or  evaluated,  or  the  source  may 
reference  other  knowledge  sources  and  leave  it  as  "an  exercise 
for  the  elicitor"  to  go  solicit  those  sources  as  well. 

The  elicitor,  at  least  initially,  is  faced  with  a  breadth 
vs.  depth  issue:  how  should  he/sne  steer  the  source  through  the 
potentially  complex  knowledge  network  and  what  is  the  potential 
impact  of  that  steerage?  Should  they  try  to  get  detail  first  or 
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try  to  form  "the  big  picture"  and  perhaps  establish  boundaries  to 
the  domain  of  interest?  How  much  knowledge  is  "enough"  or  how 
much  knowledge  remains  to  be  elicited?  How  does  the  elicitor 
know  when  he/she  is  finished? 

It  may  be  that,  since  both  depth  and  breadth  are  eventually 
required,  it  does  not  matter  in  which  direction  the  elicitation 
process  flows  but  only  that  it  eventually  come  back  to  expand 
along  all  the  different  dimensions.  Just  discovering  what  those 
dimensions  are  could  be  a  major  accomplishment  of  the  knowledge 
eliciation  process. 
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APPENDIX  B 

ELICITATION  MATERIALS  USED  IN  INTERPRETATIVE  AND 
GENERATIVE  ELICITATION  SESSIONS 


B-l 


APPENDIX  B 


CONTENTS 


Page 


B.l  Interpretative  Elicitation  Forms  for:  Scripts,  Frames, 


Mental  Models,  Analogies,  and  Critical  Incidents  .  B-5 

Scripts: 

Script  Identification  .  B-6 

Tracks  Listing  .  B-7 

Goals . B-8 

Entry  Conditions . B-9 

Results . B-10 

Roles . B-ll 

Props . B-12 

Scenes : 

Scene  List . B-13 

Description . B-14 

Actions . B-15 

Frames : 

Standard  Slots  .  B-16 

Additional  Slots  .  B-17 

Mental  Models : 

Context . B-18 

Objects . B-19 

Forces  . . B-20 

Interactions  .  B-21 

Results . B-22 

Analogy:  Reference/Target  .  B-23 

Critical  Incident  Elicitation: 

Good  Cases . B-24 

Bad  Cases . B-25 


B.2  Generative  Elicitation  Forms  for:  Production  Rules, 

Semantic  Nets,  Mental  Models,  and  Critical  Incidents  .  B-27 

Production  Rules : 

Identified . B-28 

Compartmentalized  .  B-29 

Generalized . B-30 

Chained . B-31 


CONTENTS  (continued) 


!  Page 

Semantic  Nets: 

Frame  -  Standard  Slots . B-32 

Frame  -  Additional  Slots . B-33 

Mental  Models: 

Context . B-34 

Objects . B-35 

Forces . B-36 

Interactions  .  B-37 

Results . B-38 

Critical  Incident  Elicitation: 

Good  Cases . B-39 

Bad  Cases . B-40 

B.3  Generative  Elicitation  Using  a  Specific  Problem  .  B-41 


B-3 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 

4 


B-4 


B.l  Interpretative  Elicitation  Forms  for 


Scripts: 

Script  Identification 
Tracks  Listing 
Goals 

Entry  Conditions 

Results 

Roles 

Props 

Scenes : 

Scene  List 

Description 

Actions 

Frames : 

Standard  Slots 
Additional  Slots 

Mental  Models: 

Context 

Objects 

Forces 

Interactions 

Results 

Analogy:  Reference/Target 

Critical  Jpgi^ept  .EjLicjtgtign: 
Good  Cases 
Bad  Cases 
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SCRIPT 


EXPERT* ID:  _  Pag*  _  of 


SCRIPT  IDENTIFICATION 


1.  SCRIPT:  (description) 


SCRIPT-ID:  (unique  Identifier) 
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SCRIPT 


EXPERT- ID: 


unique  identifier 


Pege  _  of 


TRACKS  LISTING 

2.  How  would  you  describe  the  [situation  development/order  of  battle] 

process  (and  please  point  out  the  important  differences  between  them  for 
low- intensity  vs.  mid- intensity  vs.  hiyh- intensity  conflict. 
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SCRIPT 


EXPERT- ID: 

SCRIPT:  _ 

TRACK:  _ 

IF  THIS  IS  THE  FIRST  TRACK  FOR  THIS  SCRIPT,  DO  3a;  OTHERWISE  JUST  REVIEW  IT. 

3a  In  doing  the  script  description,  e.g.,  order  of  battle/situation 
development,  what  exactly  is  your  objective? 

Script  Goal(s):  Refer  to  Frame(s): 


Page  _  of 


GOALS 


Refer  to  M-M(s): 


3b  Are  there  any  other  objectives  besides  the  most  obvious  one  that  you 
would  try  to  accomplish? 

Track  Goal(s):  Refer  to  Frame(s): 
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Refer  to  M-M(s) : 


SCRIPT 


EXPERT -ID: 


Page  _  of 


SCRIPT: 


TRACK: 


ENTRY  CONDITIONS 


4.  Could  you  describe  under  what  circumstances  other  than  on  order  from 

your  commander  or  when  actually  in  a  conflict  that  you  would  [begin  the 
process  of  situation  development]  [develop  an  order  of  battle]. 


ENTRY  CONDITIONS: 


Refer  to  Frame (s) : 
frame- ids 
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SCRIPT 


EXPERT- ID:  _  Page  _  of 

SCRIPT:  _ _ _ 


TRACK: 


RESULTS 


5.  What  specific  results,  or  outcomes  do  you  expect  to  get  when  you  follow 
this  procedure?  Please  describe  everything  you  can  think  of  in  the  way 
of  "results 


Results 


Refer  to  Frame(s): 


SCRIPT 


EXPERT- ID:  _  Page  _  of  _ 

SCRIPT:  _ _ 

TRACK:  _  PROPS 

3.  What  tools  (FM's;  maps,  files,  communication  devices,  automation  aids, 
etc.)  do  you  use?  Why  are  they  essential  to  your  Job? 


Props : 


Refer  to  frame(s): 


SCRIPT 


EXPERT- ID:  _  Page  _  of  _ 

SCRIPT:  _ 

TRACK: _ SCENE  LIST 


8.  (ORDER  OF  BATTLE) 

Without  going  into  a  specific  case  or  situation ,  could  you  describe  the 
general  sequence  of  actions  involved  in  developing  and  maintaining  an 
Order  of  Battle. 

(SITUATION  DEVELOPMENT) 

Without  going  into  a  specific  case  or  situation ,  could  you  describe  the 
general  sequence  of  actions  involved  in  the  process  of  situation 
development . 


Scene  No . 


Scene  Description 
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SCRIPT 


EXPERT- ID:  _  Page  _  of 

SCRIPT:  _ 

TRACK:  _  SCENE:  _ 


9a.  Considering  that  sequence  of  actions  one  at  a  time,  let's  take  [scene 

description] .  I'd  like  you  to  describe,  in  as  much  detail  as  possible, 
vhat  would  take  place.  Please  begin  by  describing  the  purpose  of  this 
particular  task. 


Subgoal: 


M-M  reference:  (Ask  "Why?"  if  not  subgoal  obviously  approp.) 
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SCRIPT 


EXPERT- ID: 


Page  _  of 


SCRIPT: 


TRACK: 


SCENE: 


Action: 


F£.eJne(s)  M-Mf  s> 

frame-ids  M-H  id(s) 
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EXPERT- ID: 


FRAME-ID: 


Frame -of : 
is-a: 

a-kind-of : 
similar- to: 
has -parts: 
part-of : 
size: 

Description: 


ms 


(unique  identifier) 


Page  _ 

FRAME 


(range) , 


(default 


Spatial -Configuration: 


_  of  _ 

STD  SLOTS 
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1 


run 

EXPERT- ID:  _ _ _ _ 

FRAME-ID:  _ _ _ _ _ 


P«g«  _  of  _ 

additional  slots 


Slot/sccrlbucm  ntat:  Attribute  lit t 
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MENTAL  MODEL 


EXPERT- ID:  _ 

MENTAL-MODEL- ID:  _ 

(unique  identifier) 


Pege  _  of 


SSHim 


1.  Why ?  (use  Chat  rule  or  procedure ) 


Context : 
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MORAL  MODEL 


EXPERT-ID:  _  Page  _  of  _ 

MENTAL-MODEL- ID:  _  OBJECTS 


2.  Please  list  for  me  all  the  personnel,  equipment,  and  any  other  things  or 
people  who  are  critical  parts  of  this  process.  If  they  have  their  own 
set  of  objectives  and  ways  of  influencing  the  process,  please  describe 
that  too. 


Object  Description  Goal ;  Farce i 

(personnel,  equipment,  etc.)  ("objectives" )  ("influence") 
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MENTAL  MODEL 


EXPERT- ID:  _  Page  _  of  _ 

MENTAL- MODEL- ID:  _  FORCES 


3.  Nov  let's  go  over  again  all  the  sources  of  influence  or  paver  that  will 
drive  this  process. 


Forces i 


Frpgsisli 

frame' ids 
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MENTAL  MODEL 


EXPERT-ID:  _  Page  _  of  _ 

MENTAL- MODEL- ID:  _  INTERACTION 


4.  Nov,  I'd  like  you  to  Cell  me  hcv  these  different  people  end/or  things 
Chet  were  identified  earlier  [refer  to  OBJECTS  list]  interact  with  one 
another  in  this  process. 


InWCtifiML 


IP  Frame  (s) 
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mental  model 


EXPERT- ID:  _  Page  _  of  _ 

MENTAL-MODEL- ID:  _ _ _  RESULTS 


5.  What  are  the  various  potential  results  of  this  process,  based  on  these 

various,  possible  interactions?  Please  describe  every  plausible  one  you 
can  think  of. 


Result(s> : 


Frame (s);  M-M(s): 
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Analogy:  Where  Target  is  analogous  to  Reference 


Reference  Doaain/System  b.  Target  Domain/System 


Features  of  Similarity:  (Complete  list;  when  done,  ask  which  of  these 
features  are  most  important  to  understanding  the  target  domain/system. 
Put  an  asterisk  (*)  next  to  those  features.) 


Important  Dissimilarities/Differences  between  reference  and  target 
domains : 


Critical  Incident  Elicitation  -  "Good"  Cases 


a.  Can  you  describe  the  features  that  would  define  an  exceptionally  good  job 
at  f developing  the  enemy  Order  of  Battle!?  What  would  define  a  really 
top-notch  performance  of  this  task? 


b.  So,  if  all  I  knew  about  the  way  an  [OB]  was  done  were  those  things  you've 
listed,  I  could  be  confident  of  having  a  very  good  product? 


c.  Can  you  think  of  an  actual  case,  in  your  experience,  that  would  il¬ 
lustrate  that  kind  of  performance? 


d.  Can  you  think  of  other  cases  that  similarly  demonstrate  a  particularly 
skillful  job  of  [developing  an  OBI? 
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Critic*!  Incident  Elicitation  -  "Bad"  Cates 


Can  you  describe  the  features  that  would  define  an  exceptionally  poor  Job 
at  f developing  the  enemy  Order  of  Battle)?  What  would  define  a  really 
poorly  executed  performance  of  this  task? 


So,  if  all  1  knew  about  the  way  an  [£)£]  was  done  were  those  things  you’ve 
listed,  I  could  reasonably  expect  that  I  would  have  a  rather  inferior 
product? 


Can  you  think  of  an  actual  case,  in  your  experience,  that  would  il 
lustrate  that  kind  of  performance? 


Can  you  think  of  other  cases  that  similarly  demonstrate  a  particularly 
poor  job  of  [developing  en  QB1? 


B.2  Generative  Elicitation  Forms  for: 


Production  Rules: 

Identified 

Compartmentalized 

Generalized 

Chained 

Semantic  Nets: 

Frame  -  Standard  Slots 
Frame  -  Additional  Slots 

Mental  Models: 

Context 

Objects 

Forces 

Interactions 

Results 

Critical  Incident  Elicitation: 
Good  Cases 
Bad  Cases 


B-27 


FIODOCTIO*  RULES 


EXPERT -ID:  _  Pag*  _  of 

PROBLEM  (level  of  detail!: _ 


Production  Rule: 


Pyle: 


HzMLs)  i  Fiemis): 

(why?)  (what?) 
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PRODUCTION  IUIU  -  eoipcrtMDUlliid 


EXPERT-ID:  _  Pag*  _  of  _ 

PROBLEM:  _ __ _  PRODUCTION  RULES 


Coopartnent  No.  sss 


Antecedent  Conditions:  (under  which  rules  below  apply) 


Consequent  Rules: 

Rule  No:  Rule:  Frame (s)  M-Mf s> 

(e.g.  rule  no  sss.tcc  where  sss  - 
compartment  no!  set  of 

antecedent  cond .  and  ttt  -  specific  rule 
no.  in  that  compartment) 


PRODUCTIWI  RULES  -  (mnlltid 


EXPERT- ID: 

PROBLEM: 

Generalized  Rule: 

IF 

THEN 

Specific  Instances  of  General  Rule:  (fill  this  first  then  get  general  rule) 
Rule  No.  Rule  Framers')  M-M(sl 

e.g.  rule  no.  xxx.jyj  where 
xxx  -  genl  rule  no  and  yjy  - 
specific  rule  no) 


_  Paga  _  of  _ 

PRODUCTION  RULES 

General  Rule  No.  sss 


PRODOCTIO*  ftUUS  -  chained 


EXPERT- ID:  _  Page  _  of  __ 

PROBLEM:  _  PRODUCTION  RULES 

_  CHAIN  NO: 


Ref:  Preceding  Rule  # _  (e.g.,  rule  no.  aaa.bbb  where 

aaa  —  chain  no.  and  bbb  -  rule 
no.  in  chain) 

Tnwslsl  M-M(s) 

(aaa) 


IF 

THEN 

v 

# 

IF 

THEN 

v 

# 

_IF 

THEN 

v 

# 

IF 

THEN 

v 

# 

_IF 
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♦/ 


EXPERT- ID: 


FRAKE-ID: 


run 

(For  Semantic  Net) 


(unique  Identifier) 


Peg*  _ 

FRAKE 


Frame -of : 


a-kind-of : 
similar- to: 


size:  '(range), 

Description: 


(default 


Spatial-Configuration: 


_  of  _ 

STD  SLOTS 
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- '  « 

* 

i 

EXPERT* ID: 


run 

(For  Semantic  Net) 


P«|«  _  of 


FRAME- ID:'- 


ADDITIONAL  SLOTS 


Sloe/seeelbue«  ns as:  sctrlbute  Use 
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MORAL  MODEL 


EXPERT* ID:  _ _  Peg® 

MENTAL-MODEL- ID:  _ 

(unique  identifier) 


1.  Why?  (use  chat  rule  or  procedure) 


Context : 
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_  of 

CONTEXT 


MENTAL  MODEL 


EXPERT- ID:  _  P*g*  _  of  _ 

MENTAL-MODEL- ID:  _  OBJECTS 


2.  Please  list  for  me  ell  the  personnel,  equipment,  end  any  other  things  or 
people  vho  ere  critical  parts  of  this  process.  If  they  have  their  own 
set  of  objectives  and  ways  of  influencing  the  process,  please  describe 
that  too. 


OMect  Description:  £a&li  EglCC  I 

(personnel ,  equipment,  etc.)  ("objectives" )  ("influence" ) 
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MENTAL  MODEL 


EXPERT-ID:  _  Page  _  of  _ 

MENTAL-MODEL- ID:  _  FORCES 

3.  Now  let's  go  over  again  all  the  sources  of  influence  or  power  that  will 
drive  this  process. 


Forces: 


Frame (s) ; 
frame- ids 
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MENTAL  MODEL 


EXPERT- ID:  _  Page  _  of  _ 

MENTAL-MODEL- ID:  _  INTERACTION 


4.  Nov,  I'd  like  you  to  tell  me  how  these  different  people  end/or  things 
thee  were  identified  earlier  [refer  to  OBJECTS  list]  interact  with  one 
another  in  this  process. 


Interactions : 


M-M-ID  Frame (s) 
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MENTAL  MODEL 


EXPERT-ID:  _ _ _  Page  _  of  _ 

MENTAL-MODEL- ID:  _  RESULTS 

5.  What  are  the  various  potential  results  of  this  process,  based  on  these 

various,  possible  interactions?  Please  describe  every  plausible  one  you 
can  think  of. 

Resylg(.s.) FrWSe(£)  ;  M-M(s) : 
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Critical  Incident  Elicitation  -  "Good"  Cases 


Can  you  describe  the  features  that  would  define  an  exceptionally  good  Job 
at  f developing  the  enemy  Order  of  Battle!?  What  would  define  a  really 
top-notch  performance  of  this  task? 


So,  if  all  I  knew  about  the  way  an  [QB]  was  done  were  those  things  you’ve 
listed,  I  could  be  confident  of  having  a  very  good  product? 


Can  you  think  of  an  actual  case,  in  your  experience,  that  would  il 
lustrate  that  kind  of  performance? 


Can  you  think  of  other  cases  that  similarly  demonstrate  a  particularly 
skillful  job  of  f developing  an  OBI? 


Critical  Incident  Elicitation  -  "Bad"  Cases 


Can  you  describe  the  features  that  would  define  an  exceptionally  poor  job 
at  [developing  the  enemy  Order  of  Battle!?  What  would  define  a  really 
poorly  executed  performance  of  this  task? 


So,  if  all  I  knew  about  the  way  an  was  done  were  those  things  you've 

listed,  I  could  reasonably  expect  that  I  would  have  a  rather  inferior 
product? 


Can  you  think  of  an  actual  case,  in  your  experience,  that  would  il¬ 
lustrate  that  kind  of  performance? 


Can  you  think  of  other  cases  that  similarly  demonstrate  a  particularly 
poor  job  of  f developing  an  OBI? 


B. 3  Generative  Elicitation  Using  a  Specific  Problem 

The  generative  approach  to  elicitation  involved  the  messages  contained 
in  the  "PERINTREP  Number  14"  and  the  "INTSUM  Number  21"  presented  in  Appendix 

C.  In  contrast  to  the  use  of  those  materials  in  interpretative  elicitation, 
the  generative  method  involved  giving  those  messages  to  the  subject  one  at  a 
time,  without  supplying  the  background  information  and  order  of  battle 
templates  from  Appendix  C.  When  messages  made  reference  to  map  coordinates, 
the  subject  was  given  blank  maps  (USACGSC  50-323  and  USACGSC  50-324)  on  which 
to  plot  information. 
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BACKGROUND 


You  are  a  replacement  G2  operations  officer  or  OB  analyst 
recently  assigned  to  the  54th  Infantry  Division.  You  have  been 
provided  some  basic  information  on  the  enemy  order  of  battle  and 
the  general  tactical  situation. 

CURRENT  SITUATION 

The  U.S.  10th  Corps  has  ordered  the  54th  Infantry  Division, 
currently  in  reserve,  to  conduct  an  attack  through  the  23rd 
Armored  Division  to  seize  the  high  ground  in  the  vicinity  of 
Neukirchen  (OBJ  1).  Based  on  the  terrain,  there  are  two  avenues 
of  approach  that  the  division  might  use.  Elements  of  the  31  GMRD 
and  49  MFD  are  defending  astride  these  avenues. 

The  enemy  forces  opposing  the  23rd  Armored  Division  are  in 
defensive  positions  and  essentially  deployed  as  depicted  on  the 
overlay. 

SPECIAL  SITUATION/TASK 

As  the  new  G2  operations  officer/OB  analyst,  you  would  by  now 
have  some  familiarity  with  the  enemy  situation  and  are  now 
prepared  to  take  appropriate  actions  in  response  to  reported 
changes  in  the  enemy  situations  as  the  information  is  provided 
from  reports  and  intelligence  summaries.  Key  to  this  exercise  is 
that  the  interviewer  will  want  to  know  what,  if  anything,  each  new 
piece  of  information  means  to  you,  as  G2  operations  officer/OB 
analyst,  and  what  actions  you  would  take. 
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10TB  (US)  COftfS  US  IODIC  DITILLXOttCB  ItfOBT  (ftSDCTSCP)  VO  II 
38CRET  For  Train  inf,  Otharvisa  Unclassified 

_  coplaa 

GOWANT 


Copy  do  _ of 

10th  (OS)  Corpa 


302130  August _ 

PERINTRIP  HO  14 

Par iod  Covered:  280500  to  290500  August 

References:  Hap,  series  USACGSC  250*158,  Germany,  sheet  1  (MUNSTER— 
WURZBURG),  edition  1982,  1:250,000  (Hap  B  (sheet  1  of  2)). 

Hap,  series  USACGSC  250-157,  Germany,  sheet  1  (MAGDEBURG— 
LEIPZIG),  edition  1982,  1:250,000  (Map  B  (sheet  2  of  2)). 

1.  GENERAL  THREAT  SITUATION 

Threat  24  CAA  defends  from  MARBURG  (MB8428)  to  G  RUN  BERG  (MB9704)  to 
FELDKRUCKEN  (NB1300)  to  DIRCHBRACHT  (NA2084)  to  RCHSTHAL  (NA2774)  to  MAR  JOB ! 
(NA3768)  with  the  31  GMRD,  49  MRD,  57  TD,  and  the  46  MRD  north  to  south. 

-  The  50  TD  has  been  confirmed  as  occupying  the  second  defense  .belt  in  thej 
24  CAA  sector. 

2.  THREAT  ACTIVITIES 
a.  Ground. 

(1)  Dissatisfaction  evident  in  57  TD  because  unit  has  been  given! 
last  priority  on  supplies.  Including  artillery  ammunition  and  rations.] 
31  GMRD  has  first  priority. 

(2)  Airphoto  reconnaissance  confirms  that  main  defense  belt  will  b< 
located  in  immediate  vicinity  of  current  line  of  contact.  No  security  zon< 
will  be  formed,  but  local  security  has  been  placed  forward. 

(3)  Movement  forward  of  additional  forces  to  support  the  SoutherJ 
Front  is  being  hampered  by  Allied  air  interdiction  and  the  overall  lack  olf 
success  of  attack  in  southern  zone. 
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ffHaOMf  14—  10tb  <W)  Corps 

(4)  50  TD  (Mtiood  defacs*  belt)  has  been  ordered  to  radio-listening 

silence  262100  August. 

(5)  Intercept  recorded  280700  August  restricted  first-echelon  MRD 
artillery  units  froa  firing  counterbsttery  firs  "unless  threat  forces*  fir* 
is  producing  asjor  loss**  of  personnel  tnd  squipoenfc." 

(6)  Above-ncraal  level  of  threat  administrative /logistics 

trsnsoisslons  were  noted  throughout  the  reporting  period. 

(7)  Movement  of  engineer  units  end  building  saterlals  forward  along 
threat  sain  supply  routes  (MSRs)  appears  to  have  Increased. 

(8)  Following  input  fro®  HtHINT,  IMINT,  and  SIG1NT  has  been 
received  during  this  period 

(a)  STRAIGHT  tLOSB  radar  identified  at  coordinates  NB388146  at 

290331  Aug. 

(b)  FAN  SONG  radar  identified  and  located  NB454131  at 

290300  Aug. 

(c)  PAT  HAND  radar  identified  at  coordinates  NB585175  at 

280600  Aug. 

(d)  PAT  HAND  radar  identified  and  located  at  NB689191  at 

290115  Aug;  previously  identified  only  as  SAH  battalion. 

(e)  FAN  SONG  radar  Identified  and  located  NB610240  at 

290513  Aug. 

(f)  STRAIGHT  FLUSH  radar  located  at  coordinates  NB440290  at 

27 0421  Aug. 

(g)  LONG  TRACK  radar  located  NB47  5  295  at  282213  Aug. 

(b)  PAT  BAND  radar  identified  and  located  at  coordinates 

NB520313  *t  290141  Aug. 

-  ( i)  Gonsunlcatloes  froa  01  SCUD  unit  NB5$5350  to  24  CAA 
observed  281935  Sep.  Unable  to  aonltor  24  CAA  coasislcatlons. 

U)  FA V  S0N0  radar  identified  and  located  at  NB610350  at 

290245  Aug;  previously  thought  to  be  Iocs  ted  NB6  335. 
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PKR2NTR1P  H~  10th  (US)  Corps 

00  LONS  TRACT  radar  locatsd  NB372442  at  29001?  Aug. 

(1)  LONS  TRACT  radar  located  NB421400  at  290043  Aug. 

(a)  LONS  TRACK  and  PAT  HAND  radars  collocated  at  NB481432  *t 

290051  Aug. 

(n)  LONG  TRACK  radar  confirmed  coordinates  NB581475  at 
290357  Aug;  previously  thought  to  be  located  NB5548. 

(o)  U1  SCUD  unit  at  NB666435  observed  comaunlcatlng  with  front 

at  281943  Aug.  Front  communications  not  observed.  ” 

(p)  U1  SCUD  bn  at  NB5444  detected  by  IMINT  at  281530  Aug. 

(q)  UI  engineer  unit  with  two  tactical  float  (FMP)  bridges  In 
place  at  NB4327  at  281530  Aug  detected  by  IMINT. 

(r)  UI  engineer  unit  at  NB4731  with  two  tactical  float  bridges 
(PMP)  at  281530  Aug  detected  by  IMINT. 

(a)  UI  engineer  unit  asseably  area  located  by  IMINT  vie  NB4215 
at  281530  Aug. 

(t)  CAA  chemical  defense  unit  located  by  IMINT  vie  NB7130  at 

281530  Aug. 

(u)  SCALEBOARD  bn  located  by  IMINT  vie  NB7447  at  281540  Aug. 

(v)  SCALEBOARD  bn  located  by  IMINT  vie  NB7433  at  281545  Aug. 

(w)  UI  airborne  unit  asseably  area,  possible  regiment  size, 
located  vie  NB9149  by  IMINT  at  281545  Aug. 

(x)  Class  I,  III,  and  V  support  area  located  by  IMINT  vie 
NB693?  at  281545  Aug. 

(y)  Class  V  support  area  located  by  IMINT  vie  NB4804  at 

28 1545.  Aug. 

(z)  UI  engr  bn  asseably  area  vie  NB9434  located  by  IMINT  at 

281550  Aug. 
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SECRET  for  Training,  Otherwise  Unclassified 
mimxr  u— ioth  (w)  corps 

(aa)  Ul  MRD  In  assembly  art*  vio  OKRA  (QB1740)  confirmed  by 
IMIirr  at  281600  Aug* 

(bb)  Class  1,  III,  and  7  support  araa  confirmed  by  IKIIIT  at 
PB2045  at  281555  Aug. 

(cc)  Class  T  support  area  confined  by  XMINT  at  PB4352. 

(9)  The  following  battalion-sited  units  are  believed  to  have  been 
rendered  coabat-lneffective  and  assimilated  into  adjacent  units  as 
replacements: 

31  GKKD 

3  CURB,  59  GMRR 
1  CURB,  120  GMRR 

3  CURB,  120  GMRR 

4  GTB,  120  GMRR 
3  GTB,  87  GTR 
32  (122-mm)  Gds  Arty  Bn, 

1  Gds  Arty  Regt 
120  (122-mm,  SP)  Gds  Arty  Bn, 

*  120  GMRR 

87  ( 122-tan,  SP)  Gds  Arty  Bn, 

87  GMRR 

57  TD 

3  TB,  167  TR 
3  MRB,  153  MRR 

b.  Air.  Threat  air  during  the  reporting  period  has  been  active.  The 
preponderance  of  available  air  sorties  has  been  used  to  establish  air 
superiority  over  the  10th  (OS)  Corps  sector  but,  at  best,  threat  air  has 
been  able  to  achieve  only  air  parity.  Although  the  24  CAA  is  not  the  main 
effort  of  the  front,  it  is  expected  that  air  support  from  the  front  air 
division  will  continue.  It  is  anticipated  that  the  24  CAA  will  receive 
approximately  four  sorties  per  hour.  The  helicopter  detachment  or  squadron 
at  division  level  will  continue  to  provide  limited  administrative  (liaison) 
support.  Attack  helicopters  in  the  helicopter  squadron  will  most  likely  be 
employed  with  the  division  counterattack  force. 

c.  Electronic  Warfare. 

(1)  Strict  S1GSEC  measures  have  been  imposed  on  all  50  TO  units. 

SECRET  For  Training,  Otherwise  Unclassified 

C-12 


49  MRD 

3  MRB,  168  MRR 
3  MRB,  172  MRR 
3  MRB,  177  MRR 
3  TB,  160  TR 
65  (122-mm)  Arty  Bn, 
38  Arty  Regt 

46  MRD 

3  MRB,' 144  MRR 

3  MRB,  146  MRR 

4  TB,  146  MRR 
3  MRB,  150  MRR 

162  (122-ma)  Arty  Bn, 
86  Arty  Regt 


38CT0  For  Training,  Otheivise  Unclassified 
FWWTMP  14—  10th  (OS)  Corps 

(2)  Borne*  ja— lng  —s'  cxp«r ienced  on  frequencies  in  th«  PM  bonds 
periodically  throughout  tho  reporting  period. 

d.  Other.  An  FM  transaitter  tentatively  located  at  NB029200  reported 
reaching  a  destination  and  requested  further  instructions.  No  response  was 
monitored  and,  after  several  attenpts  to  establish  communications,  the 
transmissions  ceased. 

3.  ANALYSIS  AND  DISCUSSION 

a.  The  24  CAA  currently  Is  defending  approximately  75  to  80  kilometers 
of  frontage.  Although  this  frontage  is  not  excessive  for  a  CAA  at  or  near 
full  strength,  the  24  CAA  is  overall  at  approximately  70-percent  strength  in 
personnel  and  equipment. 

b.  The  threat  holds  hastily  prepared  positions  along  the  current  line 
of  contact.  Although  continuing  to  improve  its  defenses,  a  severe  lack  of 
engineer  assets  continues  to  plague  its  effort. 

c.  The  57  TD  positions  are  receiving  only  light  attention.  More 
emphasis  appears  to  be  going  to  the  main  defense  belt  positions  on  the 
REISKIREHEN — ALSFELD  corridor  and  the  STEIN  AD— FULDA  corridor. 

d.  Threat  . is  capable  of  delaying  from  current  positions  eastward. 

4.  CONCLUSIONS 

The  24  CAA  will  attempt  to  defend  along  the  current  line  of  contact 
awaiting  the  outcome  of  the  major  threat  effort  in  the  north.  The  24  CAA 
remains  vulnerable  to  a  continued  corps  offensive. 

ii««*  •  * 
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PART  riVK.  EXTRACT,  23D  ARHD  DIV  INTELLIGENCE  SUMMARY  (IHTSUM)  WO  21 
SECRET  For  Training,  Otherwise  Unolasalfied 
IMMEDIATE 

FROM:  CDR,  23D  AJMD  OIV 

TO:  CDR,  10TH  (US)  CORPS 

INFO:  CDR,  3D  (GE)  Corps 
CDR,  5**TH  MECH  DIV 
CDR,  5 3D  MECH  DIV 

3NTSUM  HUMBER  21,  ENDING  302HOO  AUG  _ . 

THREE  ALFA. 

(1)  AT  1830  A  COMPANY-SIZED  FORCE,  REINFORCED  WITH  TANKS,  CONDUCTED  A 
LIMITED  OBJECTIVE  ATTACK  FROM  THE  NORTHEAST  ON  THE  LEIDENHDFER  KOPF  (MB8617). 
UNSUCCESSFUL,  THE  THREAT  FORCES  WITHDREW  TO  HASTY  DEFENSES  VICINITY  MB878I8O 
AND  MB888168. 

(2)  COMBAT  PATROLS  ENGAGED  PLATOON-SIZED  THREAT  POSITIONS  VICINITY 

MB8572  38  AND  MB861223.  AT  LEAST  ONE  TANK  WAS  DUG- IN  IN  THE  POSITION  AT 

MB857 238 . 

(3)  12  TANKS  REPORTED  IN  ONE  ASSEMBLY  AREA  VIC  WITT  ELS  BERG  (MB9023). 

(«)  A  COMBAT  PATROL  ENGAGED  A  THREAT  PLATOON  IN  A  HASTY  DEFENSIVE 

POSITION  VICINITY  OF  MB885159.  THE  POSITION  DID  NOT  APPEAR  TO  HAVE  OVERHEAD 

COVER  AND  THE  COMMUNICATIONS  TRENCHES  WERE  NOT  YET  COMPLETED.  THE  PATROL 
REPORTED  THAT  WHEN  FIRST  SIGHTED,  THE  THREAT  APPEARED  TO  BE  PLACING  MINES 
BY  HAND. 


(5)  UNITS  HAVE  REPORTED  RECEIVING  TANK  AND  ATGM  FIRES  FROM  THE  VILLAGE 
OF  RABENAU  (MB9113),  AND  FRCM  AT  LEAST  A  COMPANY-SIZE  FORCE  IN  POSITIONS  IN 
VICINITY  0FMB920122,  MB937  123,  AND  HB938114. 

(6)  REFUGEE  REPORTS: 

(A)  FORCED  TO  DIG  TRENCHES  AND  BUNKERS  IN  A  WOODLINE  1  KM  EAST  OF 
KESSELBACH  (MB9113). 

(B)  FORCED  TO  DIG  10  EX  PLACEMENTS  FOR  ANTITANK  GUNS,  PROBABLY 
100 -MM,  IN  A  WOOD  LINE  OVERLOOKING  THE  MAIN  HIGHWAY  2}  KM  SOUTHWEST  OF 
ERMEXROD  (NB0911). 
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(C)  FORCED  TO  DIG  HULL-DOWN  POSITIONS  FOR  WHEELED  ARMORED  VEHICLES 
(P  ROB  AULT  SAGGER  BRDH)  1H  THE  WOODLIKE  ON  THE  SOUTHERN  SLOPE  Of  THE 
TUMHELBERG  <MB98l1). 

(D)  CONFIRMED  A  COMPAHY-SIZED  CP  LOCATED  OK  THE  HEGBERG  (KB9709). 
BELIEVED  TO  BE  THE  1ST  CO,  1  MRB,  168  MRR,  *»9  MRD. 

THREE  ECHO.  THE  LIMITED  OBJECTIVE  ATTACK  ON  THE  LEIDENHOFER  KOPF  WAS 
Pfi£cEt)Eb  "fit  ONLY  A  SHORT  10-MINUTE  ARTILLERY  BARRAGE. 

FOUR  ALFA.  DURING  THE  PERIOD,  AN  ESTIMATED  15  KIA  WERE  REPORTED. 

FOUR  BRAVO. 

(1)  THREE  WOUNDED  PRISONERS  WERE  CAPTURED  AT  THE  LEIDENHOFER  KOPF. 
THEY  INDICATED  THAT  THEIR  PLATOON  WAS  ORDERED  TO  DEFEND  LONDORF  (HB9013). 
DOCUMENTS  FOUND  ON  ONE  OF  THE  PRISONERS  INDICATED  THAT  HE  WAS  FROM  THE  2  CO, 

2  GMRB,  59  GMRR,  31  GMRD. 

SEVEN  ALFA.  REFUGEE  REPORTS: 

(1)  A  REGIMENTAL-SIZED  CP  (BELIEVED  TO  BE  THE  172  MRR,  49  MRD)  LOCATED 
IN  RUPPERTENROD  (NB0608).  ALSO  THE  1  MRB  CP  IS  COLLOCATED  WITH  THE 
REGIMENTAL  CP. 

(2)  A  PREVIOUSLY  UNDETECTED  152-MM  GUN /HOW  BATTALION  (2/14/16?)  LOCATED 
VICINITY  OF  HOINGEN  (HB9418). 

(3)  RECONNAISSANCE  ELEMENTS  OF  THE  175  TR  WERE  REPORTED  IN  KIRTORF 
(NB0824). 

(4)  THE  86  FROG  BATTALION  (49  MRD)  LOCATED  NB1018. 

(5)  THE  87  GTR  (31  GMRD)  LOCATED  IN  NEIDERKLEIN  (MB9927). 

(6)  TANKS,  IDENTIFIED  BY  BUMPER  MARKINGS,  OF  THE  168  GTR,  50  TD,  WERE 
REPORTED  IN  EICHENAU  (NB3707). 

EIGHT  AND  NINE. 

(1)  AT  1930  A  COLUMN  OF  25  TANKS  WAS  DETECTED  MOVING  SOUTH  ALONG  THE 
MAIN  ROUTE  TOWARD  MARDORF  (MB9423) • 

(2)  AT  1935  A  COLUMN  OF  20  TANKS  WAS  DETECTED  MOVING  VEST  ALONG  THE 
AUTOBAHN  (E4)  TOWARD  EHRINGSHAUSEN  (NB0717). 
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ITSUM  21 
TWELVE. 

(1)  THE  LOCAL  ATTACE  AT  1830  APPEARS  TO  HAVE  BEEN  AN  ATTEMPT  BT  THE 
THREAT  TO  IMPROVE  HIS  DEFENSIVE  POSITION  BT  DENTING  OS  OBSERVATION  AND  PZRES 
FROM  THE  LEIDENHOFER  KOPT.  THE  SHORT  (10- MINUTE)  BARRAGE  INDICATES  THAT  THE 
THREAT  MAY  BE  EXPERIENCING  A  SHORTAGE  OP  ARTILLERY  AK40NIT10N. 

(2)  WITH  THE  EXCEPTION  OP  LOCAL  ATTACKS  AIMED  AT  IMPROVING  DEFENSIVE 
POSTURE,  IT  APPEARS  THAT  THE  THREAT  INTENDS  TO  CONTINUE  THE  DEFENSE  IN  ITS 
CURRENT  POSITIONS. 

•  •  •  •  •  ■  • 
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1.  NEW  UNIT  SIGHTING 

A  POW  from  2nd  Battalion,  84  MPR  said  that  he  had  seen  T-80 
tanks  in  an  assembly  area  near  MARDORF  (NB  9424).  He  heard  that  a 
Guards  Tank  Division  would  attack  in  the  31st  MRD  area.  The  31st 
MRD  has  T-64  tanks. 

NOTE:  Thus  information  conflicts  with  al'l  other  reports  and 
estimates  and  poses  a  good  opportunity  to  test  interviewee's 
production  rules. 

2.  EMPLOYMENT  OF  NUCLEAR  WEAPONS 

An  IMINT  report  tentatively  identifies  a  convoy  of  6 
transporters  carrying  SCUD  missiles  at  (NB  4030)  .  This  is  the 
first  sighting  of  a  possible  SCUD  battalion  resupply, 

NOTE:  Since  nuclear  weapons  have  not  been  used  up  to  this  point, 
this  report  raises  the  question  of  a  possible  nuclear  or  chemical 
attack.  Also,  this  sighting  is  forward  of  any  known  SCUD 
battalion  location. 

3.  EMPLOYMENT  OF  CHEMICAL  MUNITIONS 

10th  Corps  reports  that  a  POW  from  the  84th  MRR  said  his 
company  was  instructed  to  wear  chemical  protection  coveralls  at 
all  times.  A  second  report  identified  covered  ammunition  trucks 
with  unusually  heavy  security  delivering  munitions  to  artillery 
units  in  31st  MRD.  Some  drivers  were  wearing  protective  suits. 

NOTE:  This  is  a  first  report  on  possible  use  of  chemical  weapons 
and  does  not  fit  previous  pattern  of  intelligence  reports. 
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1.  Scripts 
Situation  Development 
Tracks:  Echelon 

Battalion  -  pointer  to  BN  subscript 
Division 

Conflict:  Low-intensity  -  pointer  to  low- 

intensity  frame 

Features:  less  technical  in  low- 

intensity  conflict 
High- intensity  -  pointer  to  high- 
intensity  frame 

Features:  more  time  sensitivity, 

higher  lethality,  more 
firepower  that  can  be 
used  quickly 

D-Main:  Pointer  to  D-Main  frame 
D-Tac:  Pointer  to  D-Tac  frame 

Goals:  Determine  enemy's  intent 

Preempt  enemy's  plan  and  operation 

Cause  initiative  to  move  to  U.S.  Commander 

Stop  enemy  from  impacting  on  friendly  forces 

Subgoal:  Kill  enemy 

Subgoal:  Target  enemy 

Obtain  knowledge  of  current  and  future 
situations  by  echelon:  battalion,  brigade, 
division 

Let  commander  know  what  enemy  is  doing 

At  D-Main:  Plan  the  next  battle  24,  28,  72 
hours  out 

Provide  commander  and  G-3  with  what 
they  need  for  their  mission: 
knowledge  of  enemy,  terrain,  and 
weather 

At  D-Tac:  Knowledge  of  current  battle  less 

than  24  hours  out 
Knowledge  of  battlefield  within 
10  km  of  FEBA 


Entry 

Conditions :  Ongoing 

Changes  in  process  driven  by 
Start  of  war 

Findings  as  a  result  of  analysis 
Enemy  actions 

Roles:  Intelligence  analyst  -  pointer  to  analyst  frame 

Support  element 
G-l 
G-4 

At  D-Main:  Commander 
G-2 

G-2  operations/situation  developer 

Assistant  G-3 

Terrain  personnel 

Staff  weather  officer 

Electronic  warfare  officer 

Collection  manager 

Collection  management  on 

dissemination  staff 

Operations  security 

At  D-Tac:  G-3 

Assistant  division  commander  - 
Operations  and  Training 

Props :  Maps 

Situation  map 
Templates 

Order  of  battle  sheets 
Gazeteer/reference  book 
Workbooks 

Imagery  products  (photos,  infrared,  radar) 

Information  reports  (from  reconnaissance 
systems,  interrogations,  SITREP  from  forward 
units) 

Analysis  of  area  of  operations 
FM  radios 
G-2  hot  loop 

Multi-channel  phone  systems 
Command  vehicle 

Results:  May  get  conflicting  information 

May  get  information  with  no  identifiable 
bearing  on  current  situation  -  pointer  to 
intelligence  analyst  frame 
May  get  information  that  does  not  support 
previous  conclusion 

Products:  Estimate  of  situation  (written  in 

All-Source  Production  section  of  D-Main) 
Intelligence  preparation  of  the 
battlefield  (IPB) 
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INTSUM  (Intelligence  Summary) 
PERINTREP  (Periodic  Intelligence 
Report)  -  done  every  6  to  12  hours 


At  D-Main:  Intelligence  reports  sent  to  D-Tac 
Spot  reports  sent  to  D-Tac 
Briefings 

Intelligence  annex  (enemy  situation 
or  forecast) 


At  D-Tac:  No  formal  reports 

Spot  or  intelligence  reports  as 
information  comes  in 
Good  result:  locate  units  and  what 
they  are  doing 


Scenes:  (Note:  Since  these  are  combined  from  more  than  one  expert,  the 

ordering  may  not  correspond  exactly  to  how  they  would  be 
performed . ) 

1)  Get  briefed  as  you  come  on  shift 

Actions:  Briefed  on  analysis  of  area  of  operations,  weather, 
terrain,  and  enemy  situation 
Read  current  plan 
Read  current  order  of  battle 
Read  current  situation  estimate 

Briefed  on  commander's  and  higher  commander's  intent 
Understand  what  own  unit  is  doing  in  terms  of  attack 
and  defense 

Assess  enemy  historical  and  current  actions  in  terms  of 
doctrine  and  departure  from  doctrine 
See  what  chemicals  have  been  used 

2)  Look  at  terrain 

Actions:  Look  at  avenues  of  approach  through  area  of  mission 
Call  terrain  analysts 

a)  Find  out  status  of  bridges  and  roads  along  avenues 
of  approach 

b)  Find  out  where  choke  points  are  and  where  enemy  can 
stop  us 

c)  Get  assessment  of  enemy  morale 

d)  See  if  there  are  indications  of  logistics  and 
resupply  problems 

3)  Call  staff  weather  officer 

Actions:  Get  forecast  for  next  3  days.  Make  sure  information 
gets  to  ASPS 

4)  Talk  to  personnel  of  division  that  is  on-line  defending  (if  any) 
Talk  to  G-2  operations 
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Vl 


Actions:  a)  Get  assessment  of  what  enemy  units  in 

b)  What  is  first  defensive  belt  doing  and  how  they  are 
doing  it 

c)  Get  assessment  of  enemy  morale 

d)  See  if  there  are  indications  of  logistics  and 
resupply  problems 

Talk  to  G-2: 

a)  See  what  threats  and  resistance  friendly  battalions 
and  brigades  will  face  as  they  pass  through  axes  of 
advance  into  enemy  territory 

5)  Talk  to  personnel  of  on-line  brigade  holding  key  avenue  of 
approach 

Actions:  Talk  to  S-2 

Talk  to  commander 

Get  assessment  of  what  enemy  units  in  first  defensive 
belt  are  doing  and  how  they  are  doing  it 
Get  assessment  of  enemy  morale 

See  if  there  are  indications  of  logistics  and  resupply 
problems 

6)  Generate  questions  that  need  to  be  addressed  for  mission 

Actions:  Get  intelligence  requirements  from  commander  (D-MA1N) 

Gl,  G3,  G4  generate  questions  as  part  of  their  plans 

7)  Go  to  all  source  production  section 
Actions:  Tell  them  what  information  is  needed 

8)  Go  to  collection  management  and  dissemination  section 

Actions:  Tell  collection  manager  what  information  is  needed  to 
ensure  that  their  information  gets  to  ASPS 

9)  Interaction  with  counterparts  at  higher  and  adjacent  units 

Actions:  Contact  by  radio  or  liaison  officers 
Share  assets 
Exchange  information 

a)  Understand  how  they  manage  collection  asst  i  so 
they  can  maximize  their  use 

10)  Conduct  limited  reconnaissance 

11)  Analysis  of  incoming  information 

Actions:  Look  at  2d  echelon  forces 
Look  at  flank  situation 

Look  at  artillery  groupings,  regimental,  and  divisional 

If  using  air,  look  for  air  defense  artillery  assets 

Look  at  current  enemy  forces 

Look  at  chemical  assets 

Look  at  what  all  divisions  are  doing 


Look  at  electronic  warfare  assets 
Look  at  engineering  assets 

a)  If  brigade  is  forward  then  planning  attack 


12)  Evaluate 

Actions:  Compare  situation  to  enemy  doctrine 

Compare  situation  to  enemy  recent  pattern  of  operations 
Have  we  looked  deep  enough? 

Determine  how  commander's  mission  can  be  supported 
Assess  enemy  strength 

13)  Brief  commander 


Track:  D-Tac 


Scenes: 


1)  Raw  information  comes  in 

2)  Post  information  relevant  to 
current  battle  on  map 

3)  Pass  information  to  D-Main 

Actions:  Decide  what  information  is 

important  to  them 

4)  Log  information  in  journal 

5)  Get  Information  from  commander 
and  G-3  on  what  to  look  for 

a)  May  include  obstacles,  key 
bridges ,  enemy  forces , 
sympathy  of  population 


Track:  Battalion  intelligence  gathering 


Goals:  Determine  force  in  contact 

Determine  enemy's  disposition 
Determine  enemy's  morale 
Determine  enemy's  intent 
Take  advantage  of  known  weakness  in 
C2  or  C3 


Entry 

Conditions:  Battalion  comes  in  contact  with  enemy 

Roles:  S2,  prisoners  (squad  leaders,  company  commanders) 

Scenes:  1)  Take  prisoners 

Actions:  a)  Use  intelligence  to  go  after  higher 
person  and  command  post  at 
battalion,  regiment,  and  higher 


Script:  Collection  Management  and  Dissemination 

Goals:  Collect  and  disseminate  information  about  enemy 

Roles:  Collection  manager,  ASPS,  commander,  G-2,  G-2 

operations 
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Props:  Radars,  radios,  photo 

Results:  Get  data  on  enemy 

Scenes:  1)  Identification  of  intelligence  requirements 

Actions:  G-2  operations  or  G-2  or  ASPS  state 
what  information  is  needed 

2)  Generation  of  collection  plan 

Actions:  Collection  manager  generates  plan 
for  collecting  data 

3)  Task  units 

Actions:  Collection  manager  tasks  units  to 
collect  data 

a)  If  right  collection  assests  do 
not  belong  to  collection 
manager ,  then  s/he  asks  next 
higher  unit  for  data 

4)  Information  comes  in 

Actions:  Receive  information  from  AV  reports, 
Corps  reports,  national  reports 

5)  Dissemination 

Actions :  When  data  comes  in  give  to  ASPS 

When  reports  come  in,  disseminate 
throughout  unit 

Script:  All  Source  Production  Section  (ASPS) 

Goals:  Analyze  intelligence  data 

Roles:  OB  tech,  OB  analyst,  analysts,  G-2,  G-2  operations, 

chief  of  ASPS,  commander,  collection  manager 

Results:  Write  estimate  of  situation 

Scenes:  1)  Identification  of  intelligence  needs 

Actions:  G-2  or  G-2  operations  tells  ASPS 
what  information  is  needed 

2)  Interact  with  CMDS 

Actions:  Tell  them  what  gaps  are  in 
intelligence 
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3)  Interact  with  terrain  analysts 

Actions:  Tell  them  what  gaps  are  in 

intelligence  regarding  terrain 

4)  Analyze  data 

Actions:  Get  electronic,  signal,  photo 

intelligence  synthesis  into  coherent 
intelligence  product 

5)  Write  estimate  of  situation 


Script:  Pre-battle  Situation  Development 

Goals:  Develop  contingency  plans 

Subgoal:  Keep  plans  up-to-date 

Entry 

Conditions:  World  events 

Political  decisions 
New  month 

Roles:  Joint  chiefs  of  staff 

Politicians 

Props:  Plans  existing 

Outcomes :  Updated  plans 

Scenes:  1)  Review  plans 

2)  Look  at  political  and  world  events 

3)  Make  inferences  regarding  potential  mission  of 
unit 

4)  Update  or  generate  plans 

2 .  Frames 
Frame:  D-Main 

Has  parts:  ASPS,  collection  management  and 

dissemination  section,  terrain 
detachment,  staff  weather  officer, 
collection  assets,  finished  intelligence 

Mission:  Plan  operations  24,  48,  72  hours  out 
Plan  next  battle 

Personnel:  G-2,  deputy  G-3,  collection  manager, 

commander 
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Frame : 


D-Tac 


Frame : 


Frame : 


Has  parts:  Raw  information 

Mission:  Run  current  battle 

Look  less  than  24  hours  out  (default  8  to  12  hours) 
Look  within  10  km  of  FEBA 

Personnel:  G-3,  assistant  division  commander  for 

operations  and  training 


Spatial 

Configuration:  Located  near  FEBA  with  brigades 


Collection  Management  and  Dissemination  Section 
Is  a:  Intelligence  unit 

A-kind-of:  Divisional  asset 

Has-parts:  Patrols;  communications,  electronic  and 

surveillance  systems,  ground  surveillance 
radars ;  airborne  and  ground  based 
platforms;  imagery  reconnaissance  (photo, 
infrared,  radar) 

Part-of:  Division  intelligence  element 
Personnel:  Collection  manager 

Mission:  Provide  intelligence  information  to  analysts 


Soviet  Offense 


Is-a:  Battle  formation 

Has-parts:  Divisions,  regiments,  artillery, 

electronic  warfare  assets,  engineer 
assets,  logistics,  nuclear,  biological, 
and  chemical  (has -mission,  seal  flanks1 
weapons 

Activity:  Movement  of  units  forward 
Firing  active 


Spatial 

Configuration:  Two  units  up,  two  back 

Division  frontage  8  to  16  km 
Artillery  forward- -2/3  range  past  FEBA 
Electronic  warfare  assets  on  flanks 
Engineer  assets  forward 


D- 10 


2nd  echelon  regiments  spaced  2  to  3 
hours  between  them 

Regimental  assets  at  rear  of  formation 
Attributes:  Combat  units  strong 

Frame :  Friendly  Offense 

Is-a:  Battle  formation 

Has-parts:  Divisions,  terrain,  avenues  of  approach, 

objectives,  air  and  ground  forces,  enemy  units 

Spatial 

Configuration:  Divisions  are  within  boundaries 

Avenues  r f  approach  run  through 
division  boundaries 

Objectives  (has -part  terrain)  are  phased 
Enemy  on  other  side  of  FEBA  -  pointer 
to  enemy  defense  frame 

Frame:  Soviet  Prepared  Defense 

Frame  of:  Defensive  operation 

Is-a:  Battle  formation 

Has-parts:  Combat  units,  headquarters,  telephone 

wires,  radios,  artillery,  overhead  cover, 
communication  trenches,  mine  fields, 
engineer  units 

has-part:  building  materials 

Activities:  Enemy  digging  in 

Lots  of  artillery  used 
No  forward  movement 

Spatial 

Configuration:  Layers:  Screen  of  platoons  forward 

First  defense  belt 
a)  Supporting  elements  to 
rear  and  adjacent 
Second  belt 
Third  belt 

Antitank  or  counterattack 
force  in  2nd  or  3rd  belt 
Reinforcements  behind  2nd  belt 

Attributes:  Combat  units  weak 


Frame:  Hasty  Defense 

Frame  of:  Defensive  operation 
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Frame: 


Frame : 


Is-a:  Battle  formation 

Has-parts:  Air  defense  (mission:  protect  command 
posts,  logistics  sites,  bridges), 
artillery,  command  posts,  logistics  sites, 
bridges 

Mission:  Hold  off  enemy  attack  until  own  2d  echelon 
attack  can  be  reformed 


Spatial 

Configuration:  Division  frontage  15  to  25  km 

Artillery  forward- -2/3  range  past  FEBA 
Air  defense  assets  near  command  posts, 
logistics  sites,  bridges 

Chemical  Warfare 

Frame  of:  Fighting  strategy 

Is-a:  Warfare 


Has-parts:  Friendly  and  enemy  units,  chemicals 

Results :  Impedes  battle  conditions 

Initiated  by:  Friendly  stronger  than  enemy 


High  Intensity  Conflict 

Frame  of:  Battle 

Is-a:  Conflict 

Description:  Europe 

Mechanized  and  armored  warfare 
Large  numbers  of  forces  involved 
Full  range  of  weapons 

Spatial 

Configuration:  Battlefield  characterized  by  width  and  depth 
No  halts  or  breaks  in  battlefield 

Attributes:  Weapons- -accurate,  lethal 

Contrasts  with:  Low- intensity  conflict 

Comparative 

Features:  Weapons  less  technical  in  low- intensity  conflict 

Support  procedures  less  correlated  in 
low- intensity  conflict 
Analytical  effort  constant  in  both 
More  firepower  in  high- intensity  conflict 
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Frame : 


Person 
Frame : 


Person 

Frame: 


More  destruction  brought  quickly  in 

high-intensity  conflict 

High- intensity  more  dynamic 

Know  what  enemy  will  do  early  and  with 

what  and  where  in  high- intensity  conflict 


Low  Intensity  Conflict 
Frame-of:  Battle 

Is-a:  Conflict 

Has-parts:  Weapons,  support  procedures 

Opponent:  Third  world  military  force 

Features:  Weapons --not  sophisticated 

Support  procedures- -not  sophisticated 


Contrasts  with:  High- intensity  conflict 
Comparative 

Features:  Weapons  more  technical  in  high- intensity  conflict 

Support  procedures  more  correlated  in  high- intensity 
conflict 

Analytical  effort  constant  throughout  both 


G-2  Operations 

Tasks:  Supervises  intelligence  gathering  and 

production 

Knowledge:  Not  necessarily  expert  on  enemy 

Knows  where  to  get  information 
a)  Knows  capability  of  information 
gathering  systems 


Good  Intelligence  Analyst 

Knowledge:  Knows  enemy  tactics,  capabilities,  and 
limitations 

Capabilities:  Can  avoid  self  deception 
Keeps  an  open  mind 
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3. 


Mental  Model:  Surprise  Attack 

Context:  Battlefield 

Goals:  Seize  objectives  before  enemy  can  respond 

Cause  disarray  in  enemy 

Objects:  Friendly  units 
Enemy  units 

has -part:  supporting 
has-part:  first  echelon 

Forces:  Combat  forces 

Mobility 

Force/Object 

Interactions:  Friendly  force  impacts  enemy  force 

Friendly  force  uses  intelligence  to  target 
obstacles  to  movement 

Friendly  force  destroys  obstacles- -pointer  to 
M-M  destroy 

Outcomes:  Friendly  forces  advance  impeded 

Enemy  cannot  react  quickly 
Friendly  forces  achieve  objectives 


Mental  Model:  Cooperation 

Context:  General 
Current 

Instantiation:  Exchange  of  corps  and  division  intelligence 
information  on  battlefield 

Goals:  Assist  commander's  mission  and  conceptions 

(missions  have  property  compatible) 

Objects:  Information 

Corps  and  division  collection  assets 
Analysts 

Forces:  Common  goals 

Common  understanding 

Force/Object 

Interactions:  Division  commands  has  mission 
Corps  commander  has  mission 
Division  commander  and  staff  have 
information  relevant  to  corps  commander's 
mission 

Corps  commander  and  staff  have  information 
relevant  to  division  commander's  mission 
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Division  commander  and  staff  understand 
division  commander's  mission 
Corps  commander  and  staff  understand 
division  commander's  mission 
Division  commander  and  staff  give 
information  to  corps  commander  and  staff  and 
ask  them  for  information 

Corps  commander  and  staff  give  information 
to  division  commander  and  staff 

Outcomes:  Corps  and  division  commanders  and  staff  have 

information  they  need 

Corps  and  division  commanders  and  staff  have 
broader  assessment  and  understanding  of 
situation 

Corps  and  division  commanders  can  better  achieve 
their  missions 

Relationships  between  corps  and  division 
commanders  and  staffs  have  been  improved 

Mental  Model:  Self  Deception  About  Enemy 

Context:  Battlefield  (universal) 

Coal:  Understand  enemy 

Objects:  Intelligence  analyst 
Information 
Enemy  forces 

is-a:  Soviet 

or  is-a:  Third  World  Force 
Force:  Reasoning  process 

Force/Object 

Interactions:  Reason  that  third  world  country  have 
primitive  capabilities 

Reason  that  Soviet  forces  are  overly  strong 

Outcomes:  Develop  misco  :opt  which  result  in  loss  of  life 

on  battlefield 


Mental  Model:  Disrupt  Enemy  Forces 

Context:  Battlefield  (universal) 

Goal:  Kill  enemy  before  he  can  take  action  that  would 

impact  on  friendly  forces 

Objects:  Enemy  forces 

(Friendly  forces) 

Enemy  commander 

Friendly  commander 

Friendly  intelligence  officer 
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Forces:  Combat  power 

(Reasoning  power) 

Force/Object 

Interactions:  Friendly  forces  strike  before  enemy 
forces  contact 

Outcomes:  Enemy  plan  disrupted 

Enemy  train  of  thought  disrupted 
Enemy  intention  disrupted 
Enemy  force  destroyed 

Enemy  forces  do  not  impact  on  friendly  forces 
Initiative  goes  to  friendly  commander 


B.  Situation  Development:  Generative  Elicitation  Methods 
1 .  Production  Rules: 

a.  Tactics/Doctrine 

If  enemy  is  in  hasty  defense, 

then  they  will  be  going  into  attack  mode  or  they  will 
wait  for  reinforcements  and  they  will  not  be  very 
long 

If  enemy  attack  is  reinforced  with  tanks, 
then  objective  of  the  attack  is  important  or  the  attack 
is  a  supporting  one  and  more  force  is  needed  to 
defeat  its  enemy 

If  enemy  has  insufficient  organic  force  to  conduct 
successful  attack, 

then  it  will  attach  other  forces  to  the  fighting  force 

If  the  commander  is  a  brigade  level, 

then  he  needs  to  know  about  company  size  forces 

If  commander  or  intelligence  analyst  is  at  an  echelon, 
then  he  needs  to  know  about  composition  of  units  that  he 
is  fighting  and  composition  of  flanking  units  and 
composition  of  rear  units 

If  commander  or  intelligence  analyst  is  at  brigade, 
then  s/he  needs  to  know  about  composition  of  units  that  s/he 
is  fighting  and  composition  of  flanking  units  and 
composition  of  rear  units 

If  unit  is  infantry  and  it  is  to  be  reinforced, 
then  it  will  be  reinforced  with  tanks 

If  unit  is  tanks  and  it  is  to  be  reinforced, 
then  it  will  be  reinforced  with  infantry 
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If  echelon  Is  brigade  and  enemy  conducts  a 
company-size,  limited  objective  attack, 
then  brigade  personnel  need  to  know  what  happened  on 
the  flanks  of  that  company 

If  enemy  is  conducting  an  attack, 

then  size  of  the  attacking  force  is  proportional  to  the 
size  of  the  objective 

If  a  battalion  uses  only  one  company  to  attack, 
then  it  may  be  a  limited  objective  attack 

If  unit  is  tank  battalion, 

then  it  will  not  reinforce  its  companies  with  tanks 

If  enemy  battalion  is  not  tank, 
then  it  may  be  motorized  rifle 

If  goal  is  to  reinforce  subunits  and  unit  is 
regiment, 

then  take  tank  company  from  tank  battalion  and  give  it 
to  motorized  rifle  battalion 

If  goal  is  to  reinforce  subunits  and  unit  is 

motorized  rifle  battalion  and  it  has  been  given 
tank  company  by  regimental  commander, 
then  break  up  tank  company  and  give  platoons  to 
motorized  rifle  companies 

If  enemy  on  defense, 

then  2/3  artillery  range  will  be  inside  own  lines 
If  attack  has  limited  objectives, 

then  will  be  preceded  by  artillery  barrage  of  10  to  15 
minutes 

If  attack  is  major, 

then  has  artillery  barrage  of  30  to  60  minutes 

If  force  is  attacking, 

then  it  needs  force  advantage  of  3:1  to  6:1 
and  it  is  weighted  with  tanks 
and  it  is  supported  by  artillery 

If  force  is  meeting  engagement, 
then  needs  3:1  for  a  ratio  to  win 

If  enemy  forces  are  spread  out, 
then  there  is  not  enough  strength  to  defend  along 
frontage,  or 

there  is  not  enough  time  to  display,  or 

the  enemy  does  not  expect  effort  and  that  avenue 

of  approach 
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If  enemy  bringing  up  material  to  dig  and  having 
trouble  resupplying  at  least  one  front, 
then  counterattack  likely 

If  Soviets  in  defense, 

then  friendly  attack  was  more  successful  than  their 
attack 

If  Soviets  in  deliberate  defense, 

then  look  for  counterattack  force  within  the  next  24  to 
36  hours 

If  battalion  CP  is  collocated  with  regimental  CP, 
then  the  unit  is  not  combat  effective  and  is 
consolidating  CPs, 
or  the  2d  echelon  has  moved  up, 

or  the  exploitation  force  or  counterattack  force  is 
being  held  back 

If  the  enemy  has  three  segments  on  line, 

then  the  enemy  is  not  expecting  a  friendly  attack  In 

strength, 

or  enemy  attack  was  a  supporting  attack  since  the 
forces  were  not  concentrated 

If  it  is  a  tank  division, 

then  the  tank  force  is  used  for  reconnaissance 

If  unit  is  a  motorized  rifle  regiment, 
then  tanks  are  used  for  exploitation  force 

If  overhead  cover  is  used  and  if  the  nuclear  capable 
units  are  moving  forward, 

then  the  enemy  may  be  ready  to  use  nuclear  weapons 

If  a  Soviet  column  of  tanks  is  moving  forward, 
then  a  reinforcing  force  may  be  present 

If  a  defense  is  established  without  security  zones 
and  defensive  belts, 

then  the  defense  is  short  term  in  preparation  for  a 
counterattack 

If  defense  is  hasty, 

then  mines  may  be  emplaced  by  hand 

If  defense  is  deliberate, 

then  mines  may  be  emplaced  mechanically 

If  the  objective  is  terrain, 

then  the  attack  may  be  more  spread  out 

If  tanks  are  dug  in, 

then  the  threat  is  defending 
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If  troops  are  in  an  assembly  area, 
then  something  may  be  about  to  happen 

If  the  forces  are  concentrated  in  an  area, 
then  it  is  likely  to  be  a  main  attack 

If  the  enemy  restricts  fire  of  its  artillery  unit, 
then  the  enemy  is  trying  to  avoid  being  detected  or  the 
enemy  wants  to  save  them 

If  a  company  is  straight  infantry  or  straight 
motorized, 

then  enemy  action  is  less  important 

If  ribbon  bridge  is  emplaced  by  enemy, 
then  enemy  is  preparing  to  attack 

b.  Orpanization/Eoulpment 
If  report  is  about  companies , 

then  infer  that  unit  getting  report  is  brigade  level  and 
sender  of  report  is  battalion  S-2 

If  person  is  battalion  S-2, 

then  s/he  will  get  reports  on  squads  and  platoons 
If  fighting  in  Germany, 

then  opponent  is  Warsaw  Pact,  probably  Soviet 

If  echelon  is  division, 

then  staff  are  concerned  with  battalions 

If  unit  is  Scale  Board  Battalion, 
then  target  is  corps  and  above 

If  two  tank  regiments  are  moving  in  parallel  at  the 
same  time, 

then  they  are  likely  to  be  a  single  tank  division 
If  enemy  on  defense, 

then  they  will  use  wire  more  than  radio 
If  enemy  is  on  defense, 

then  1/3  of  SCUD  range  should  be  in  front  of  enemy 
front  lines 

If  the  unit  is  reinforced  with  A, 
then  the  unit  is  not  of  type  A 

If  adjacent  units  are  both  type  A, 

then  a  higher  level  unit  is  type  A 

If  adjacent  units  are  different  types, 

then  the  higher  level  unit  can  be  either 
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If  the  number  of  enemy  tanks  equal  12, 
then  It  is  likely  to  be  a  tank  company 

If  there  are  antitank  guided  missiles, 
then  the  enemy  is  an  infantry  force 

If  enemy  vehicles  are  wheeled  armored  vehicles, 
then  the  unit  is  a  motorized  rifle  unit 

If  SA2,  SA4,  and  SA6  system  radars  are  present, 
then  an  enemy  army  is  involved 

If  SA2  radars  are  present, 
then  enemy  level  is  Front 

If  SA4  radars  are  present, 
then  enemy  level  is  Front  or  Army 

If  SA6  radars  are  present, 
then  Enemy  level  is  division 


c .  Other 


If  grid  coordinates  are  given  for  a  unit, 
then  assume  they  are  the  center  of  mass  for  that  unit 
and  that  the  unit  may  be  dispersed  around  that 
center 

If  coordinates  are  given  for  a  company, 
then  the  company  might  be  spread  out  around  a  500  meter 
radius  from  the  coordinates  reported 

If  goal  is  to  determine  dispersion  of  an  enemy  unit, 
then  look  at  flanking  units 

If  battle  lasts  4  to  5  days, 

then  not  many  people  will  be  left  to  operate  tanks  for 
deliberate  defense 

If  enemy  digging  in  and  using  refugees  to  dig  and 
their  attacks  have  limited  objective, 
then  their  strength  is  significantly  low 

If  heavily  guarded  installations  are  near  the  FROG 
unit  and  if  they  are  displayed  forward, 
then  a  nuclear  attack  may  follow 

If  a  unit  has  main  attack  responsibility, 

then  usually  this  unit  gets  first  priority  in  supplies 

If  there  are  two  salients  (one  into  their  territory 
and  one  into  ours)  and  they  did  not  attack, 
then  we  attacked 
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If  there  is  radio  silence, 
then  perhaps  enemy  on  defense 

If  engineer  units  and  building  materials  are  moving 
forward, 

then  deliberate  defense  is  likely 

If  unit  is  antitank  battalion, 
then  it  needs  trenches  and  bunkers 

If  an  area  is  getting  higher  priority  of  emergency 
and  building  materials, 
then  greatest  threat/risk  is  in  that  area 

If  unit  constructs  boundary, 

then  they  should  be  recognizable  pieces  of  terrain 
If  terrain  is  heavily  treed, 

then  the  terrain  is  not  good  for  tanks  to  maneuver 

If  the  objective  is  a  city, 
then  the  attack  is  more  canalized 

If  the  tanks  are  clean, 

then  it  is  probably  a  fresh  unit  (passage  of  lines) 

If  there  are  good  natural  boundaries, 
then  it  is  likely  to  be  a  unit  boundary 

If  the  terrain  is  broken  up, 
then  enemy  movement  will  be  slow 

If  civilians  are  doing  military  tasks, 
then  the  soldiers  are  doing  something  else  or  the  enemy 
strength  is  low  and  speed  is  essential  in  putting 
in  defensive  positions 

If  engineer  units  and  building  materials  are  moving 
forward, 

then  supply  routes  have  increased 


2 .  Semantic  Networks 

HASTY  DEFENSE 

has  property 
temporary 


COMBAT  FORCE 


is  a j 


is 


a 


motorized  infantry  tank 
rifle 
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r 

type 

tank 


type 

motorized 

rifle 


ENEMY  UNIT 

I 

type  level  level 

battalion  company  division 


DIVISION 

has-part 

regiment 

has-part 

battalion 

has-part 

company 

TANK  FORCE 

has-part 

antitank  guided  missiles  (quantity:  0) 


has- 

part 

commo 


has-  has  - 

part  part 

lh  d  .1 

overhead  mines 


trenches  coverage 


—  DELIBERATE  DEFENSE- 

I  I 

has-  has-  has- 

part  mission  part 

I  I  I 

anti-tank  hold  hull  down 
guns  ground  position 


has-property 


has- 

part 

security 

zone 


has  - 
part 


defens ive 
belts 


dug- in 


j - HASTY 

has-part 

prepared 

overhead 

cover 

(quantity:  0) 


DEFENSE 


mines 


commo 

trenches 

(quantity: 


0) 
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motorized  rifle  division 
part-of 


ANTITANK  BATTALION 


has -part 
100 -mm  tank 
guns  (qty :  12) 


has -part 
batteries 
(qty : 2) 


ARMY  DIVISION 


MOTORIZED  RIFLE  COMPANY 

„  I  .  I 

has -part  units  has -configuration 

(quantity:  4)  2  units  up 

2  units  back 


r~ 

has-part 


tanks 

(qty: 30-31) 


-TANK  BATTALION 


has -miss ion 

[ 

reinforcing 

force 


has -miss ion 

counterattack 

force 


has-mission 

I 

attack  force 
(mode  neg) - 
means  not  this 
mission 


j _ COMBAT - - 

has-mission  has-mission 


neutralize 
enemy's  ability 
to  complete 


kill  enemy 
(mode-neg) 


i.  Sexless 

Script:  Order  of  Battle 

Tracks:  1)  Defense  with  loss  of  assets 

2)  Weather  (cold  and  heat) 

3)  NBC  environment 

4)  New  information  with  no  order  of  battle  action 

a)  get  more  sources  of  information  to  confirm 

5)  No  information 

Actions:  a)  Use  doctrine 

b)  Use  local  populace  (embassy  defense 
attache) 

c)  Use  interrogators 

d)  intercept  transmissions 

Goals:  Inform  commander  whom  he  is  facing,  what  threat  is 

made  up  of,  tactics,  capabilities,  how  Soviet 
commander  is  going  to  fight. 

Suggest  weaknesses  and  ways  to  beat  enemy  (e.g., 
psychological  operations  to  get  masses  on  friendly 
side,  use  of  firepower). 

Advise  commander  on  how  to  employ  his  systems  so 
as  to  get  Soviets  to  respond  in  way  advantageous 
to  friendly. 

Personal  Goals 

of  OB  Tech:  1)  Be  first  to  know  about 

critical  information  and 
conclusions 

2)  Be  able  to  piece  parts  of 
puzzle  together 

General 

Plan:  Integrate  pieces  of  intel  that  are  reported  to 

produce  coherent,  doctrinally  consistent  picture. 
Request  collection  of  information  of  critical  and 
uncertain  facts. 

Entry 

Conditions:  Bring  to  battle:  good  working  knowledge  of  Soviet 
doctrine,  tactics,  training,  equipment,  commander 
personalities,  basic  weaknesses  and  vulner¬ 
abilities,  knowledge  of  U.S.  systems,  tactics 
friendly  commander's  style  and  information 
requirements 

In  Battle:  OB  done  ongoing 

New  information  triggers  process 
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Props: 


Humint,  slgint,  photo  info,  technical  info, 

OB  handbook,  communication  links,  telephones 
FM  radios,  overlays  templates  (situation, 
event,  decision  support,  terrain),  maps, 
microfix,  RATT  rig,  SB-22  switchboard 

Roles:  ASPS  staff  (OB  tech,  NCO  in  charge,  chief  of  ASPS) 

TCAE  (technical  control  and  analysis  element) 

CEWI  BN 
G-l  NCOIC 
G-2  NCOIC 
G-3  NCOIC 
G-4  NCOIC 
G-5  NCOIC 

Fire  support  element  of  division  artillery  (has 
function:  targeting) 

Air  Force  weather  personnel 
Terrain  detachment 

Chemical  and  biological  radiological  element 
Cavalry  unit 

Results:  Situation  report;  intelligence  report;  four 

paragraphs  in  FEBA,  enemy  disposition,  activities 
and  expected  enemy  courses  of  action;  situation 
map  on  overlay 

Scenes:  1)  When  come  on  shift  hear  briefing  presented  by 

outgoing  shift  leader 

Actions:  Focus  on  major  units  and  expected 

enemy  courses  of  action 

2)  Study  map  and  situation 

Actions:  Look  at  composition  and  disposition  of 

enemy  forces 
See  how  close  to  FEBA 
See  if  barriers  constructed 
Look  for  artillery  units  (RAGs,  DAGs) 
i)  Look  at  relation  to  FEBA- -if  close 
then  offense  indicator 
See  what  is  near  objective 

3)  Conduct  terrain  analysis 

Actions:  Look  for  highways  into  area  where 

own  unit  will  be  operating 

i)  See  where  enemy  can  come  from 

4)  Use  order  of  battle  to  fill  in  missing  data 

Actions:  Look  at  symbols  for  units  on  maps 

i)  If  none  are  there,  then  get 

information  from  OB  about  units 
not  posted  guess  where  units  might 
be  based  on  doctrine,  type  of 
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unit,  and  terrain  (e.g.,  BMP 
better  off -road  than  BTR) 


5)  Identify  gaps  in  information  and  prioritize 

Actions:  See  what  information  you  do  not 

have  (look  for  gaps) 

If  some  force -element  components  present 
(sufficient  to  imply  a  particular  size 
of  unit  but  not  all), 

then  look  for  element  missing  (e.g.,  if 
identify  three  rifle  regiments,  where  is 
4th?  where  is  artillery  CP? 

In  general,  the  absolute  size  of  piece 
or  value  of  piece  if  missing  information 
is  not  the  most  crucial  goal  for 
prioritizing  collection.  Value  relative 
to  success  of  friendly  commander's  goal 
is  primary. 

If  collecting  information  for  targeting, 
the  relative  importance  of  an  object 
would  matter,  e.g.,  main  enemy  CP  a  good 
target. 

Independent  of  commander's  goals,  other 
characteristics  that  prioritize 
collection: 

1)  Nuclear  and  chemical  operations 
units,  deployments 

2)  Artillery  positions-has  property-can 
suppress  friendly  antitank  weapon 

Generate  hypothesis  about  what  informa¬ 
tion  might  be  missing  (i.e.,  what 
"truth"  is). 

Generate  assessment  of  information 
needed  to  test  hypotheses. 

Prioritize  information  by  how  critical 
it  is  to  determining  how  enemy  can 
impact  on  commander's  plan. 

Look  at  information  requests  that  have 
been  fulfilled  and  those  still  requiring 
tasking. 

6)  Develop  event  templates 

7)  Develop  named  areas  of  interest  (NAI) 

Actions:  Take  information  requirements  and  look 

at  roads  enemy  can  use,  especially 
crossroads  where  enemy  can  change 
direction. 
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Look  for  moving  targets 

Look  at  bridges  for  enemy  crossing 

Look  at  avenues  of  approach 

8)  Request  information  from  collection  manager 

Actions:  Prioritizing  collection  of  enemy 

location  according  to  what  enemy 
forces  are  close  enough  to  pose 
immediate  threat  to  enemy  forces. 

Look  at  enemy  2nd  echelon  coming 
to  reinforce  (maneuver  elements, 
lead  elements,  artillery,  radars 
(location  ID)). 

See  if  FROGs  prepare  to  attack 
See  if  enemy  prepared  defensive 
positions . 

See  if  tank  units  attacking  or 
moving  forward. 

See  if  enemy  clearing  obstacles; 
indicates  attack  ID  of  divisions 
facing 

9)  Process  incoming  information  and  screen  same 

Actions:  When  report  comes  in  see  if 

relevant  to  targeting  -  pointer  to 
Fire  Support  subscript 

a)  If  SCUD  then  go  to  FSE,  also 
tell  G-3  and  G-2 

Assign  log  number 

Glance  through  to  see  if  immediate 
value  to  fighting 

a)  If  regiment  is  moving, 

notify  officer  in  charge  of  ASPS. 

Determine  whether  to  post  on  map  or 
other  filing  system. 

a)  Criteria  for  not  posting/ 
rejecting  report  info.: 

o  High  volume/low  information 
value  data  causes  clutter, 
confuses  SITN  map  unnecessarily 

o  No  special  interest  in 
reported  activity 

o  Force  level  reported  [more  than 
two  levels  down  or  one  level 
up]  and  [activity  reported  was 
expected] 
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b)  Process  reports  for  range  of  force 
levels  of  concern  to  OB  analyst  on 
two  levels  down  and  one  level  up 

c)  Other  Information  to  post  to  files 

o  In  addition  to  info  reporting 
uncertain  or  unexpected  enemy 
activity  post  new  info 
concerning: 

1.  biographical/personal 
factors ,  enemy 
commanders 

2 .  enemy  vehicle 
characteristics 

3.  enemy  weapon  system 
characteristics 

This  information  goes  to  common 
posting  area  (reference  files) 

d)  Posting  to  Situation  Map: 
Assessment  of  implications  of  data 
for  the  picture  on  the  map  done 
before  posting  made.  Does  new  data 
make  map  more  informative,  e.g., 
change  the  picture/than  without 
it?  If  not,  do  not  post  it. 

Look  for  data  previously  missing 

Write  notes  on  report,  e.g.,  whether 
report  is  useful,  where  to  post,  etc. 

If  reports  do  not  make  sense  then  re- 
check  with  source. 

10)  Prioritize  Information  to  Analyze 

Actions:  Prioritize  by  what  can  kill  friendly 

forces  or  leave  them  in  a  vulnerable 
position  (e.g.,  split  forces)  or  what 
assets  of  enemy  (composition  and 
disposition)  can  keep  unit  from 
performing  mission 

a)  consider  nuclear  capable  forces 
first 

b)  consider  chemical  warfare  second 

c)  airborne  forces 

d)  artillery  systems 

e)  missile  systems 

f)  maneuver  forces 
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11)  Analyze  Information 

Actions:  If  report  comes  in  which  contradicts 

expectations  of  enemy  doctrine,  then 
report  may  be  inaccurate  (source  wrong 
or  deceived)  or  report  man  be  wrong 
resolution  and  collect  information  to 
resolve  conflict. 

Do  predictive  and  situational  analysis 
in  cycles. 

Use  situational  analysis  to  verify  and 
validate  predictions  of  predictive 
analysis . 

Don't  take  Soviet  doctrine  as  immutable, 
they  may  change  way  they  do  things  if 
doctrine  doesn't  work. 

12)  Write  INTREP  every  six  hours 

13)  Make  sure  reports  are  disseminated  through  commands 
to  Corps  BDE,  BNs,  Companies,  DISCOM 

14)  Deliver  briefing 

Actions:  If  don't  know  something,  be  honest  about 

it. 

Report  uncertainty  to  commander  but  give 
worst  case  that  commander  should  prepare 
for. 

15)  Update  information  requirements  and  new  taskings  for 
commands 

a)  Develop  new  NAI's 


Subscript:  "Hypothesis  Formation" 

Hypothesis  formation  Is  part  of  the  process  of  updating  the  current 
OB.  The  OB  analyst  forms  hypothesis  about  enemy  CA  and  from  these 
generates  information  requirements  to  refute/confirm  hypothesis.  See 
Main  OB  Process. 

Entry 

Conditions:  Have  current  enemy  OB  and  situation  map  with 

current  available  reports  posted 

Have  generic  knowledge,  or  access  to  basic 
information  about  how  an  enemy  would  attack 

Props:  Situation  Map 

INTEL  Reports 

OB  Handbooks  reference  in  enemy  composition, 
behavior 
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ACT  1: 


Look  at  how  much  emphasis  the  enemy  is  placing  in 
a  given  sector  of  the  battlefield;  i.e.,  the  troop 
strength,  and  massing  of  forces,  as  well  as 
composition  of  forces. 

If  force  strength  at  location  x  appears  to  be 
increasing,  then  location  x  may  be  the  location  of 
main  attack. 

ACT  2:  Look  for  potential  objectives  (this  is  a 

refinement  of  analysis  of  ACT  1)  in  hypothesized 
area  of  attack. 

2.1  Use  knowledge  of  terrain  to  determine 
potential  objectives 

2.1.1  Do  terrain  analysis;  i.e.,  identify 

"key  terrain,"  choke  points,  decision 
points . 

Since  current  reports  are  obsolete,  mentally 
project  where  they  (the  enemy)  is  going,  and 
discard  information  that  had  only  immediate 
benefit  (and  would  now  be  obsolete). 


Subscript:  OB  for  Fire  Support 

Entry 

Conditions:  OB  analyst  has  a  basic  understanding  of 
priority  of  targets;  generic  to 
objectives.  It  is  up  to  fire  support 
to  decide  whether  to  commit  asset. 

Props  for  OB  for  fire  support/targeting: 

HUMINT  has  property  ususally  not  timely  enough  to  be  used 

SIGINT  has  property  usefulness  for  causes  artillery 

targeting  diminishes 
in  minutes 

implies  has -property 

How  long  was  artillery  moves  every 

there  before  it  got  1-1/2  hours 

reported? 


Actions:  Whenever  a  report  comes  in,  do  a 

cursory  examination  of  it  to  determine 
whether  it  has  value  for  targeting. 
Timeliness  of  intelligence  is  critical 
to  accurate  targeting  fire  support. 

If  valuable  for  targeting,  call  fire 
support  and  pass  along  the  information. 
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Script:  Soviet  Attack  and  Defense 

Tracks:  1)  Type  of  battle 

a)  offense 

b)  defense 

2)  Type  of  terrain 

a)  open 

b)  wooded 

c)  hills  and  mountains 

d)  marsh 

e)  snow 

f)  rivers 

g)  soft  sand  desert 

3)  Type  of  unit 

a)  motorized  rifle 

b)  tanks 

c)  artillery 

Track:  Offense 

Scenes/ 

Actions:  1)  Artillery  prepares  attack  to  keep 

enemy  down,  closed  up  and  on  defense 

a)  shoot  at  command  post, 
personnel  and  logistics 

b)  if  attacked  by  artillery,  move 
and  look  for  best  terrain,  or 

c)  try  to  knock  out  artillery  or 
elements  that  control 
artillery,  or 

d)  shoot  own  artillery  at  enemy 
artillery,  or 

e)  destroy  preplanned  obstacles 
along  avenues  of  approach 

2)  Tanks  move  forward  to  exploit 
through  direct  fire 

a)  tanks  shoot  at  other  tanks 

3)  Motorized  rifle  cleans  up  after 
tanks 

a)  fights  smaller  resistance 

Track:  Motorized  Rifle/Offense 

Scenes/ 

Actions:  1)  Battalion  task  force  leads  attack 

with  tanks 

2)  Infantry  fighters 

a)  normally  fight  dismounted 
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Track: 


Track: 


Tracks : 


b)  if  enemy  strong  or  in  NBC 

environment  or  light  defense 
then  fight  mounted 

3)  Radio  silence 

4)  Jamming  friendlies 

Tank/Offense 

Scenes/ 

Actions:  Tanks  lead  as  exploitation  force 


Artillery/Offense 

Scenes/ 

Actions:  Enemy  assigns  division  artillery  to 

regiment  artillery  is  placed  forward  and 
used  to  augment  firepower  fire  for  50-60 
minutes  before  division  size  attack 


Offense/Open  Terrain 

1)  No  effect  on  battle 

Offense/Wooded  Terrain: 

1)  Enemy  goes  in  long  columns  or  across  wider 

frontage  as  approaches  attack 

2)  Tanks  harder  to  attack  so  use  dismounted 
motorized  rifle  troops 

3)  Motorized  rifle  may  precede  tanks 

4)  Artillery  less  effective 

Offense/Hills  and  Mountains: 

1)  Slows  things  down 

Offense/Rivers : 

1)  Require  engineer  assets  -  pontoon  boats  or 
bridges 

Offense/Marsh: 

1)  Bypass  marsh 

Offense/Snow; 

1)  Slows  things  down 

Offense/Soft  Sand  Desert: 

1)  Slows  things  down 

2)  heat  could  cause  fatigue  in  troops 

3)  heat  could  cause  equipment  to  breakdown  faster 

4)  lack  of  water 
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Track : 


Defense 


Scenes/ 

Actions:  1)  Enemy  organizes  companies  and 

platoons  and  strong  points 

2)  Enemy  sets  up  minefields,  obstacles, 
and  barriers 

3)  Enemy  tries  to  entrap  friendly 
forces  in  kill  zones  and  fire  zones 

a)  artillery  exploits  fire  zones 

4)  Tanks  counterattack 


Track:  Motorized  Rifle/Defense 

Scenes/ 

Actions:  Organize  around  strong  points  of  platoon 

or  company  size 

Tanks  placed  behind  motorized  strong 
points  to  break  through  and  allow 
motorized  rifle  to  continue  fighting. 

Track:  Tank/Defense 

Scenes/ 

Actions:  Tanks  are  to  rear  of  strong  points  as  a 

breakthrough  force 

Track;  Artillery/Defense 

Scenes/ 

Actions:  Artillery  is  placed  back 


Tracks:  Defense/Open  Terrain: 

1)  Enemy  digs  in  more 

Defense/Wooded  Terrain: 

1)  Units  spread  out  more,  maybe  in  squads 

2)  Troops  hide  in  trees 

3)  Harder  for  overlapping  field  of  fire 

Defense/Hills  and  Mountains: 

1)  Ideal  because  can  use  reverse  slope,  pointer 
to  M-M  Reverse  Slope 

2)  Friendly  place  on  other  side  of  hill 

3)  Artillery  shot  at  higher  angle  and  move 

forward 

Defense/Rivers : 

1)  Defense  with  river  in  front 

2)  Shoot  enemy  as  they  cross  river 
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Defense/Harsh : 

1)  Not  defend  marsh  unless  it  is  around  a  city 

2)  Defense  area  near  marsh  to  force  enemy  into 

marsh 

Defense/Snow: 

1)  Coldness  may  affect  troops  and  equipment 

2)  Slows  things  down 


2 .  Frames 

Frame:  Order  of  Battle 


Function:  Order-of-battle  analysis 

Echelon:  Division 


Part-of:  G2  inteligence  function 

Has -Staff:  OB  Technician  has  number:  1 

is -type:  warrant  officer 

has -property :  100%  time  dedicated 

to  production  of  OB 

OB/Analyst  has  number:  1 

is -type:  NCO 

has -property:  25-50%  time 
dedicated  to  production  of  OB 


Has -Acronym:  OB 

Has -Input:  Knowledge  of  enemy  force  doctrine 

enemy  tactics 
enemy  training  methods 
enemy  equipment 


Frame:  knowledge  of  equipment 

has -part:  knowledge  of  vehicles 
has -property : mobility 

has -part:  knowledge  of  weapon  systems 
has -part:  weapon  use 
has -part:  fire  range 
has -part:  fire  angle 
has -part:  obliquity 

knowledge  of  enemy  compensabilities 

enemy  force  strength 

EW,  commune iations  (radio) 

enemy  force  composition 

enemy  force  disposition 
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Frame: 


Head  OB  Analyse 

Has-prop:  much  predictive  analysis  done  before  conflict  begins 
Has -prop:  read  all  new  info  on  enemy  forces 


Frame : 
Case : 


Frame : 


Has-prop:  connects  current  info  to  past  instance,  draws 
relationships,  analogies 

Has-prop:  knows  where  to  find  info  needed 

Has -training:  combat  arms 

Has -knowledge:  enemy  facing,  Soviet  OB,  area  going  to  fight, 
unit  mission,  collection  assets,  other  staff 
members,  type  of  unit  fighting,  enemy  unit 
history 


OB  Products 


Frame:  OB  Products 


In- garrison 


Case:  In-combat 


has -part:  black  books 
has -part:  cable  book 
has -part:  Intel  bulletins 
has -part:  special  reports 


has -part: 
has -part: 
has -part: 
has -part: 
has-part: 
has-part: 


INTSUMs 
PERINTREPs 
spot  reports 
OB  overlay 
situation  map 
Intel  journal 


OB  Analysis 

Has-part:  situational  analysis 

has-part:  Intel  on  enemy  past  action 
has-part:  enemy  doctrine 
has-part:  enemy  personalities 
has -goal:  assess  enemy  intent 

assess  what  enemy  is  doing 
has -role:  support  commander's  decision  making 
validate  predictive  analysis 
is -a:  continuous  cycle,  alternating  with  predictive 
analysis 

Has-part:  predictive  analysis 

has-part:  enemy  weaknesses 

has-part:  C^  vulnerabilities 
is-a:  continuous  cycle:  alternating  with  situational 

analysis 

has-part:  intel  on  enemy  action 
has -role:  support  commander's  decision  making 
guide  intel  collection  request 
has -goal:  identify  opportunities  to  exploit  enemy 
weaknesses 

has-goal:  predict  what  enemy  will  do 


Frame : 


Enemy  Force  Strength 


Format:  Force  strength  tables  In  OB  flies  and  OB  cards 

IF  you  receive  Intel  report  of  enemy  forces  destroyed 
THEN  you  record  it  in  your  OB  file  &/or  OB  cards 

Source:  Enemy  doctrine 

Records  of  cummuluative  enemy  losses 


Frame:  Enemy  Disposition 

Is -a:  Enemy  location 

Has-part:  Last  location,  current  location,  predicted  location 


Frames  of  Components  of  OB  Analysis  Frame 


Frame : 


Enemy  (Soviet)  doctrine 


Is-a: 


Description  of  the  set  way  an  army  will  employ  its 
personnel  and  equipment 


Used- for:  Training  forces  for  operation  under  wartime  conditions 

Has -prop:  Probably  will  be  modified  based  on  actual  wartime 

experiences 

Has -Source:  Soviet  newspapers,  Soviet  military  publications, 
monitoring  Soviet  exercises 


Frame :  Enemy  Tactics 

Similar- to:  Enemy  doctrine 

Is-a:  Internal  mechanism  enemy  really  follows  to  achieve  goals 

prescribed  by  doctrine 


Frame :  Enemy 

Similar-to: 
Has -prop: 
Used-for: 


Training 
Enemy  fighting 
Rigidity 

Assessment  of  combat  effectiveness, 
vulnerabilities 


enemy  force 


Frame:  Soviet  Force:  Weaknesses/Potential  Vvulnerabilities 

Part-of:  Intelligence  info  on  Soviets 
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Requires  extensive  use  of  traffic  controllers 

Troops  below  senior  level  officers  are  not  trained 
to  read  maps 

Disrupt  troop  movement  by  interdicting  traffic 
control  communication  with  higher  HQ,  and 
communication  between  traffic  controllers 

Mental  Model  Mental  Model 

interdiction  C  process 


Frame:  Soviet  Order  of  Battle 

Frame-of:  Fighting  force 

Has -part:  Composition,  tactics,  doctrine 


Frame:  Soviet  Defensive  Formation 

Has-parts:  Tanks,  divisions,  artillery,  radars,  second  belt, 

security  zone,  engineers,  building  supplies, 
motorized  rifle  units,  division  CP 

Description:  Lack  of  activity 

Spatial 

Configuration:  Tanks  back 

artillery  back  -  2/3  range  to  FEBA 

second  belt  division  command  post  15  km  from  FEBA 

radars  back 

wide  frontage 

motorized  rifle  units  up 

security  zone  forward 

engineer  and  building  supply  forward 

Compared- to:  Hasty  defense 

has-part:  security  zone  (mode  neg.) 

Frame:  Soviet  Offensive  Formation 

Has-parts:  Artillery,  radars,  supplies,  ammunition,  food,  POL, 

logistics,  second  echelon 

Spatial 

Configuration:  Artillery  up  -  1/3  range  to  FEBA 

logistics  15-25  km  from  FEBA 
second  echelon  up  -  less  than  15  km  from  FEBA 
radars  up 
narrow  frontage 


Has  - vulnerabil ity : 
Has -source: 

Has -implications : 
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Frame 


Frame 


Frame 


Frame 


Regimental  Artillery  Group  (RAG) 


Has -parts: 

Battalions  (quantity:  3) 

Part-of : 

Regiment 

Spatial 

Configuration:  Regimental  cp  in  center 

2 BN  up,  1BN  back 
in  center  of  radar 

BTR 

Frame -of: 

fighting  machinery 

Is-a: 

armored  vehicle 

A-kind-of : 

tank 

Similar- to: 

BMP 

Has-part: 

armor 

has -prop:  thin  skinned 

Part-of: 

infantry 

Compared- to 

BMP 

armor  thinner  in  BMP 

less  mobile  than  BMP,  operates  off  roads  worse 
more  easily  destroyed  than  BMP 


Soviet  Forces 

Has -part:  Combined  Arms  Army 

has-part:  motorized  rifle  division  (quantity: 3) 

has-part:  artillery  regiment 

has-part:  anti-aircraft  regiment  (quantity:  1) 
has-part:  tank  regiment  (quantity:  1) 
has-part:  MRR  (quantity:  3) 

has-part:  Howitzer  BN 

has-part:  BMPs 
has-part:  motorized  rifle 

battalions  (quantity:  3) 
has-part:  BMPs 

has-part:  tank  division  (quanity:  1) 

has-part:  motorized  rifle  regiment 
has-part:  tank  regiment  (quantity:  3) 


Chemical  Forces 

Has-part:  Persistent  chemical  agent,  non-persistent  agents 

Has-part:  Spetznaz  forces,  air  aerosol  airborne  agents 
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Frame 


Frame 


Frame : 


Has -prop:  Train  with  chemicals 

Spatial 

Configuration:  Forward  and  at  flanks  of  main  force 


Technical  Information 

Has -part :  Enemy  equipment  info 

has -source:  EAC  intel 

Has -part:  Enemy  weapon  system  info 

has - s  our ce :  EAC 


OB  Handbook 


Has -part:  Info  general  on  enemy  army  organization 

instance:  MRR 

has-part:  MRBs 


Has -part:  Specific  unit  info;  about  personnel  in  specific  armies 

and  divisions 

instance:  commander  MRD  x  is  y 


Has -part:  Biographic  info 

has-characteristic:  collected  at  EAC 
has-characteristic:  division  commander  can't  collect 

it  per  se 

has-characteristic:  division  may  get  from  POW 


HUMINT 

Is -a:  intel  reports  from  any  source 

Has-part:  PW  report 

has-part:  enemy  biographies 
source:  Brigade 
echelon:  Division 


Has-part:  IRR  agent  debriefs 

preprocessed:  No/None 
reliability:  low 

transmission  rate:  generally  low/variable 
source/accountability:  little/none 
special  handling 

instructions:  calibrate  individual  agent's  reliability 
accuracy  if  possible 
IF  want  to  calibrate  agent, 

THEN  you  need  to  get  more  than  one  report 
from  that  agent 
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Frame : 


SIGINT 


Frame 


Is -a: 


Intel  report 


Has -part:  ELINT 

is-a:  report 

has-a:  electronic  signal  identification 

instance-of:  radar  type 


Has -part:  Enemy  radar  deployment  rule 

part -of:  enemy  doctrine 

has-charac:  usually  accurate 
has-charac:  don't  accept  as  gospel 
has-charac:  accidental  or  purposeful  deception 
instance-of:  march  in  formation 

produces  simultaneous 
&  mixed  emission  of 
different  signals 
instance-of:  wrong  identity  or 

interpretation  of  signal 
data 


Has -part:  Most  radars  are  for  a  single  purpose 

part-of:  radar's  characteristics  reflect  its  mission 

e.g.:  gun  dish  radar  different  from  land  radar 


PHOTINT 


Is-a:  Intel  report 

Has -type:  Photos 

source :  AF  reconnaisance 
has-charac:  not  typical 


Has  -  type :  SLAR 

is-a:  side-looking  airborne  radar 
is-a:  given  number  of  "still  shots"  in  a  defined 
interval  of  space 


Has -type:  Moving  target  indicator 

a-kind-of:  radar  info 

is-a:  vehicle (s)/object(s)  moving  @  >  5mph 
has-charac:  high  data  volume 
has-charac:  info  value/unit  data  is  low 
has-charac:  low  resolution;  can't  distinguish 

hostile  vehicles  from  neutral/friendly 
vehicles 

handling  instructions:  in  general,  MTI  info  not 

worth  recording 
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Mental  Model: 


Reverse  Slope 
Context:  Battle  fought  on  hills 

Goal:  Friendly  force  wants  to  destroy  enemy  force 

Objects:  Hills,  enemy  force,  friendly  force 

Forces :  Friendly  and  enemy  combat  power 

Force/Object 

Interactions:  friendly  force  hide  behind  hill 

outcome:  enemy  can't  see  them 
enemy  comes  to  top  of  hill 
outcome:  enemy  in  friendly  line  of  size 
friendly  attack  enemy  force 

Outcome:  enemy  force  destroyed 

Mental  Model:  Why  Soviets  Keep  Chemical  Agents  on  Their  Flanks 

Context:  Battlefield 

Goal:  Protect  own  forces  from  destruction  (destroy  friendly 

forces) 

Object:  Terrain,  persistent  chemical  agent,  friendly  forces, 

Soviet  forces 

Forces:  Friendly  and  enemy  combat  power,  lethality  of  chemical 

agents 

Force/Object 

Interactions:  Maneuver  forces  advance 

If  chemical  agents  in  front  and  units  get 
exposed,  then  will  die 

To  avoid,  enemy  keeps  chemicals  on  flank  away 
from  units 

Outcome:  Enemy  units  not  exposed  to  chemicals 

Mental  Model:  Force  C  process 

Context:  Battlefield 

Objects:  1)  Commander  of  forces  HQ  (goal:  initiate  communication 

of  commands  for  action  and  operations;  terminal  point 
for  communication  of  current  status  of  operations  from 
troops) 

2)  Communication  nodes  (goal:  receive  command  messages 
from  previous  node(s),  subsequently  transmit  messages 
to  other  nodes) 
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3)  Communication  links  (goal:  carry  message  from  trans 
mitter  node  to  receptor  node) 

4)  Troops/command  (forces)  (goal:  execute  commands 
received  from  HQ) 

Forces:  Electronic,  optical,  auditory  transmission/reception  of 

information  for  command  and  control 


Force/Object 

Interactions:  Command:  HQ  link  to  node  to  link  to  troops 

Control:  troops  to  link  to  node  to  link  to  HQ 

Outcome:  If  all  components  function  properly,  then  command  and 

control  effective,  else  C^  ineffective  and  confusion 
results 


Mental  Model:  Friendly  Interdiction  of  Enemy 


Context:  Battlefield 


Goals:  Friendly  force  wants  to  destroy  or  significantly  reduce 

enemy  force  capability  to  carry  our  planned  operation. 
Enemy  force  wants  to  carry  out  intended  processes  and 
operations . 

Object:  Friendly  force 

Enemy  force 

Forces:  Friendly  combat  power 

Enemy  combat  power 


Force/Object 
Interactions : 


IF  friendly  force  successful, 

THEN  friendly  goal  achieved,  enemy  goal  prevented. 

IF  friendly  force  not  successful, 

THEN  enemy  goal  achieved,  friendly  goal  not 
achieved. 


B.  Order  of  Battle:  Generative  Elicitation  Methods 
1.  Production  Rules: 

a.  Tactics/Doctrine 
If  terrain  is  wooded, 

then  use  motorized  rifle  units  for  cover  and  concealment 

If  unit  is  tanks, 
then  use  an  open  terrain 
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If  enemy  is  going  into  prepared  defense, 
then  he  will  set  up  defense  belt  and  security  zones  and 
barricades  and  will  probably  not  see  counterattack 

If  enemy  has  a  security  zone, 
then  it  is  more  difficult  to  attack 

If  enemy  is  not  using  security  zone  on  defense, 
then  he  may  not  have  enough  troops  or  they  are  changing 
their  doctrine  in  the  way  they  attack 

If  enemy  is  using  security  zone  on  defense, 
then  main  defense  belt  moves  forward 

If  enemy  not  using  security  zone  on  defense, 
then  use  IPB  template  to  see  what  kind  of  offensive  or 
defensive  situation  enemy  will  use 

If  unit  is  going  into  a  hasty  defense, 

then  he  will  set  up  no  security  zone  and  he  will  set  up 
local  security  and  2d  echelon  will  not  be  ready  to 
be  committed  and  he  will  counterattack 

If  enemy  2d  echelon  is  ready  to  be  committed, 
then  enemy  will  not  go  into  hasty  defense 

If  enemy  is  committing  additional  forces, 
then  place  being  committed  is  enemy's  main  effort 

If  enemy  is  going  from  hasty  to  permanent  defense, 
then  will  leave  security  forces  in  contact, 
then  withdraw  main  forces, 
then  withdraw  security  forces 

If  Soviets  having  lack  of  success  and  2d  echelon  is 
having  problems, 

then  Soviets  may  introduce  chemical  and  nuclear  weapons 
to  regain  initiative  and  momentum 

If  there  are  no  indicators  of  chemical  or  nuclear 
attack  and  enemy  not  having  success  and  second 
echelon  having  trouble  being  committed, 
then  continue  to  look  for  indicators  of  chemical  or 

nuclear  attack  and  brief  that  there  is  no  evidence 

If  main  effort  is  having  lack  of  success, 
then  expect  follow-on  forces  to  be  shipped 

If  follow-on  forces  get  shipped  somewhere  else, 
then  main  effort  changes 

If  enemy  having  success  in  an  area, 
then  will  ship  forces  to  that  area 

If  unit  goes  or  radio  listening  silence, 
then  it  is  probably  getting  ready  to  attack 
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If  Soviets  are  going  to  attack  joing  with  2  units, 
then  they  typically  will  be  2  tank  units  or  2  motorized 
rifle  units 

If  unit  is  in  rear  and  getting  priority  of  assets, 
then  it  will  join  battle  first 

If  a  unit  goes  on  radio  listening  silence, 
then  it  may  attack  and  friendlies  can't  pick  them  up  on 
radio  anymore 

If  enemy  attacking  in  one  area  and  defending  in 
another, 

then  he  is  probably  shifting  his  effort 

If  enemy  restricts  its  artillery  fire, 
then  he  is  probably  running  low  and  won’t  be  resupplied 
for  a  while 

or  may  not  have  target  locations 

or  front  line/major  forces  can  defeat  friendly  without 
counterbattery 

If  front  is  having  lack  of  success  and  is  running  low 
on  ammunition, 

then  it  will  go  on  defensive 

If  enemy  is  not  having  success, 
then  it  is  a  good  time  to  counterattack 

If  enemy  is  going  to  attack, 

then  may  send  reconaissance  element  forward 

or  jam  friendly  communications 

or  use  smoke 

or  have  river-crossing  elements  forward 
If  enemy  is  attacking, 

then  they  will  bring  bridging  assets  foward 

If  enemey  is  reinforcing, 
then  look  for  counterattack 

If  unit  on  defense, 

then  they  will  bring  engineers  forward  to  build 

barricades  in  front  of  its  position  and  to  dig 
defensive  tank  ditches  and  set  up  mine  fields 

If  engineers  have  building  materials, 

then  they  are  probably  going  to  build  barricades 

If  enemy  is  attacking, 

then  he  will  fire  field  artillery  first 
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If  friendlies  can't  monitor  enemy  transmissions, 

then  enemy  may  be  on  radio  listening  silence 
or  range  of  equipment  might  not  be  great  enough 
or  terrain  may  block  transmissions 

or  may  be  encrypted 

If  entity  is  a  potential  target  and  unit  has  weapons 
to  attack  it, 

then  entity  is  of  interest  to  unit  as  target 

If  entity  is  within  area  of  operations  or  area  of 

interest  of  unit, 
then  it  is  of  interest  to  unit 

If  Soviets  are  at  war, 

then  the  objectives  are  to  advance  30-50  km/ day 

If  unit  defends  fixed  site, 

then  it  is  unlikely  to  move  short  distances 

If  SCUD  unit  communicated  several  hours  ago  and  no 
attack  has  been  launched  yet, 
then  communication  was  probably  not  an  attack  order 

If  air  defense  is  located  by  river, 

then  it  is  probably  protecting  river  crossing  site 

If  enemy  is  on  defense, 

then  he  will  blow  up  bridges 

If  Soviets  wish  to  destroy  friendly  aviation  assets, 
then  they  will  strike  ground  assets  with  missiles 
and  use  surface  to  air  missiles  to  his  airborne  elements 
and  use  air  to  air  tactics 

If  range  of  unit  is  short, 

then  assembly  area  must  be  close  to  target 

If  signal  security  placed  on  unit, 

then  it  may  be  getting  ready  to  attack 

If  enemy  is  engaging  in  barrage  jamming, 
then  it  is  an  attack  indicator 

If  enemy  brings  chemical  defense  forward, 

then  he  probably  will  use  it 

If  enemy  is  going  to  use  chemical  assets, 
then  he  will  bring  artillery  assets  forward 

If  enemy  airborne  unit  is  In  battlefield, 
then  enemy  will  not  use  nuclear  weapons 
If  enemy  conducts  a  limited  objective  attack  and 
retreats , 

then  he  is  probing  for  friendly  weaknesses 
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If  enemy  finds  weaknesses  in  friendly  force, 
then  he  will  lead  attack  through  that  weakness 


If  enemy  conducts  an  attack  with  battalion, 
then  it  is  a  more  major  attack 

If  locate  forward  units  of  bigger  unit, 
then  infer  attack  will  occur  where  forward  units  are 
going 

If  enemy  force  moving, 

then  direction  of  movement  may  indicate  where  attack 
may  come 

If  enemy  is  defending, 

then  frontage  is  wider  than  if  attacking 

If  enemy  is  defending, 

then  command  posts  and  artillery  are  farther  back  from 
FEBA  than  an  offense 

If  enemy  is  in  transition  from  offense  to  defense, 
then  units  may  still  be  forward 

If  tanks  are  dug  in, 

then  enemy  is  in  defensive  position 

If  enemy  is  going  to  attack, 

then  he  will  precede  attack  with  artillery  barrage 

If  enemy  used  short  artillery  barrage, 
then  it  is  probably  for  defense 

If  air  assault  unit  is  in  staging  area, 
then  it  will  be  deployed  shortly 

If  enemy  has  5  divisions, 

then  they  probably  will  place  them  3  up  (MR,  T,  MR)  and 
2  back 

If  friendly  is  attacking, 

then  need  to  assess  percentage  of  losses  that  can  occur 
from  indirect  fires 

If  goal  is  to  reduce  enemy  strength  prior  to  attack, 
then  use  indirect  fire 


b.  Organlzation/Eouipment 

If  unit  is  designated  "Guards", 
then  it  is  Russian 

If  enemy  not  Soviets, 

then  not  have  chemical  and  nuclear  weapons 
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If  goal  is  to  assess  likelihood  of  nuclear  and 
chemical  attack, 

then  need  to  know  locations  of  surface-to-surface 

missiles,  masking  activities,  and  locations  and 
activities  of  nuclear  capable  weapon  systems  or 
aircraft 

If  unit  talks  to  higher  unit, 

then  it  is  probably  subordinate  to  that  unit 

If  identify  new  unit, 

then  use  order  of  battle  to  see  who  parent  is 

If  army  running  low  on  counterbattery  artillery, 
then  can't  give  any  to  subordinate  divisions  --  pointer 
to  M-M  supply -and- demand 

If  friendly  unit  is  corps, 
then  it  will  face  an  army 

If  goal  is  to  establish  FEBA, 

then  find  out  where  forward  units  are  and  draw  line  there 

If  U.S.  fighting  a  battle, 

then  they  will  have  an  operations  order 

and  know  mission 

and  area 

and  have  maps 

and  have  mission  statement 

and  have  done  preliminary  intelligence  analysis  on 

area  of  operations  and  avenues  and  probable  enemy 
will  be  facing 

and  will  have  collection  plan 

and  have  intelligence  requirements  of  the  commander 

If  road  is  an  autobahn, 
then  it  is  good  for  tanks 

If  road  is  good  for  tanks, 

then  they  probably  will  be  placed  there 

If  Soviets  are  not  being  successful, 

then  infer  that  they  are  using  up  their  forces 

If  Soviets  using  up  their  forces, 

then  infer  they  will  bring  in  their  next  echelon 

If  subunits  dispersed  according  to  how  far  one  would 
expect  to  see  in  one  larger  unit, 
then  infer  that  they  probably  form  that  larger  unit 

If  unit  is  tank, 

then  it  will  fight  tanks  with  tanks 

If  unit  is  motorized  rifle, 

then  it  will  fight  tanks  with  anti-tank  weapons 
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If 

then 

If 

then 

If 

then 

If 

then 

If 

then 

If 

then 

If 

then 


If 

then 

If 

then 

If 

then 

If 

then 

If 

then 

If 

then 

If 

then 

If 

then 


If 

then 

If 

then 

If 

then 


information  concerns  radars , 

it  does  not  differentiate  types  of  division 

SCUD  is  communicating  to  army, 
assume  it  is  subordinate  to  that  army 

air  defense  radar  is  being  moved, 
it  will  move  more  than  3  kms 

radar  is  detected, 

look  within  a  5  km  radius  to  confirm  location 

division  has  engineering  assets, 

probably  won't  lend  them  to  another  division 

Scale  board  is  detected, 

it  is  part  of  Soviet  force  only 

Soviets  send  out  reconnaisance  elements, 
main  body  of  division  will  be  15-30  km  behind 
regiment 

FROG  moves  forward, 

it  may  be  ready  to  fire  an  offense 

goal  is  to  guess  how  many  divisions  on  defensive, 
divide  frontage  by  doctrinal  frontage  per  division 

unit  is  air  assault, 

troops  brought  in  by  helicopter 

unit  is  airborne, 

troops  parachute  to  destination  from  airplane 
enemy  is  on  defense, 

he  will  suffer  few  casualties  than  offense 

artillery  is  over  120mm, 
it  is  nuclear  capable 

Soviets  are  running  a  combined  arms  army, 

they  will  reinforce  motorized  rifle  units  with  tanks 

enemy  is  on  offense, 

he  will  have  Operational  Maneuver  Group  (OMG) 
on-line 

enemy  is  on  defense, 
he  will  not  have  OMG 

enemy  Js  preparing  counterattack, 
he  may  have  an  OMG 

enemy  is  going  to  counterattack, 
he  will  use  tanks 
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If  troops  have  guards  honor, 

then  they  may  think  they  will  get  special  priority 

If  unit  gets  first  priority  of  supplies, 
then  they  will  probably  head  main  attack 

If  one  division  has  a  better  commander  than  another, 
then  it  might  get  priority  of  supply  from  Army 
commander 

If  unit  is  required  for  mission, 

then  they  will  be  given  priority  of  supplies 

If  unit  gets  last  priority  of  supplies, 
then  they  are  leading  less  important  supporting  attack 
or  they  are  2d  echelon 

or  they  are  in  some  kind  of  supporting  role 

If  troops  not  given  support  when  fighting 
then  morale  may  be  low 

If  unit  is  going  to  attack, 

then  it  will  give  first  priority  of  supplies  to 
main  units  and  commit  them  to  1st  echelon 

If  unit  is  going  to  attack, 

then  will  give  last  priority  of  supply  to  supporting 
units  and  commit  them  to  2d  echelon 

If  enemy  is  not  getting  supplies, 
then  it  is  weak 

If  enemy  is  weak, 
then  try  to  destroy  him 

If  facing  an  enemy  unit, 

then  ask  collection  manager  for  information  regarding  it 

If  enemy  has  no  reinforcements  and  he  is  going  on 
defensive , 

then  we  are  winning 

If  enemy  is  having  difficulty  in  moving  forces, 
then  friendly  is  winning 

If  need  information, 
then  go  to  collection  manager 

If  friendly  forces  hit  Soviet  logistics  site, 
then  they  will  have  trouble  supplying  great  numbers 
of  forces 

If  enemy  has  a  lot  of  forces, 

then  supplying  them  will  be  a  slow  process 


If  enemy  logistics  are  destroyed, 
then  enemy  morale  will  be  hurt 

If  on-line  unit  is  not  getting  priority  of  supplies 
and  another  unit  is  getting  ready  to  attack, 
then  second  unit  may  be  replacing  front  line  unit 

If  front  is  having  lack  of  success  and  is  running  low 
on  ammunition  and  is  attrited, 
then  they  are  no  immediate  threat 

If  more  than  average  logistics  communications  are 
being  recorded, 

then  enemy  may  be  having  resupply  problems 
or  enemy  is  reinforcing 

If  unit  is  not  getting  supplies, 

then  may  initiate  logistics  communications 

If  enemy  communicates  too  often, 

then  they  can  be  picked  up  by  friendly  communication 

If  friendlies  intercept  enemy  communications, 
then  they  can  learn  about  their  weaknesses 

If  not  know  whether  units  are  tank  or  motorized  rifle, 
then  look  for  assets  belonging  exclusively  to  one  or  the 
other 

If  find  asset  associated  with  only  one  kind  of  unit, 
then  infer  it  is  that  kind  of  unit 

If  SCUD  is  communications  with  higher  unit, 
then  it  may  be  requesting  a  target 
or  set  up  location 
or  getting  ready  to  move 

If  terrain  is  mountainous, 
then  may  block  communications 

If  weather  is  rain  or  snow, 
then  may  block  communications 

If  entity  is  a  potential  target  and  unit  does  not  have 
weapon  to  attack  it, 

then  it  is  important  for  order  of  battle 
If  entity  is  important  to  mission, 

then  it  is  important  to  intelligence  analysts  and  others 

If  Soviets  have  not  advanced  far, 
then  they  are  not  meeting  their  objectives 

If  have  identified  two  different  command  post 
locations , 

then  can  draw  unit  boundary  between  them 
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If  Information  is  old  and  unit  referenced  is  mobile, 
then  pull  information  off  map 

If  information  is  old  and  unit  references  is  fixed 
site, 

then  leave  it  on  map 

If  entity  is  nuclear  capable, 
then  put  it  on  target  list 

If  target  is  nuclear  capable, 
then  it  is  high  priority 

If  measuring  equipment  is  not  totally  accurate  and 
object  being  measured  is  not  likely  to  change  in 
small  ways , 

then  assume  small  correction  in  measurement  is  due  to 
original  error  in  measurement 

If  higher  commander  is  talking  to  lower  commander  and 
enemy  is  preparing  to  attack, 
then  lower  commander  may  be  leading  the  attack 

If  enemy  is  short  of  manpower  or  being  cruel, 
then  may  have  refugees  work  for  them 

If  friendly  is  attacking, 

then  need  to  hit  enemy  indirect  fires 


2 .  Mental  Models 


Mental  Model :  Effect  of  enemy  nuclear  weapons  on  enemy  operations 


Context: 
Goal : 
Objects : 
Forces : 


Battlefield 

Destroy  friendly  forces 

Friendly  forces,  enemy  forces,  nuclear  weapons 
Explosive  power,  residual  contamination 


Force/Object 

Interactions:  Nuclear  weapons  destroy  friendly  forces 

area  is  contaminated 
If  enemy  goes  through,  get  killed 
enemy  must  go  around  area 

Outcomes:  Enemy  movement  hampered 

Friendly  forces  destroyed 


Mental  Model:  Using  Motorized  Rifle  Unit  in  Surprise  Attack 

Context:  Battlefield 
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Goal: 


ACCack  friendly  forces  without  detection 


Objects:  Trees,  enemy  tanks,  enemy  foot  soldiers,  friendly 

forces,  friendly  sensing  capabilities 

Forces:  Noises,  light  (sight) 

Force/Object 

Interactions:  Foot  soldiers  hide  behind  trees 

outcome:  trees  block  sight  of  them  from  friendly 
Enemy  uses  fewer  tanks 

outcome:  creates  less  noise 
Sensing  units  not  pick  up  enemy 

Outcome:  Enemy  attacks  without  detection 

Mental  Model:  Influence  of  Terrain  on  Attack 

Context:  Battlefield 

Goal:  Enemy  wants  to  take  objective  in  friendly  force  area 

Objects:  Enemy  force,  friendly  force,  terrain 

Forces:  Enemy  combat  power,  friendly  combat  power,  movement 

capabilities  of  troops  and  vehicles 

Force/Object 

Interactions:  Enemy  has  capability  to  move  toward  friendly  forces 

terrain  may  have  features  which  block  movement 
capability 

Enemy  redirects  movement  based  on  obstacles 
outcome:  enemy  is  in  different  location 
Enemy  in  wrong  place  to  attack  friendly  objectives 

Outcome :  Enemy  attacks  new  location 

Mental  Model:  Economy  of  Communication  on  Battlefield 

Context:  Battlefield 

Goal:  Enemy  units  want  to  communicate  to  other  enemy  units 

Friendly  units  want  to  destroy  enemy 
subgoal:  locate  enemy 

Enemy  does  not  want  to  be  destroyed 

subgoal:  do  not  be  detected  by  friendly 

Objects:  Enemy  units,  enemy  parent  units,  other  enemy  units, 

friendly  units,  friendly  communication  devices, 

enemy  communication  devices,  friendly  weapons 
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Forces: 

Force/Object 
Interactions : 


Mental  Model: 

Context : 

Goals : 

Objects : 

Forces : 

Force/Object 
Interactions : 

Outcomes : 


Friendly  combat  power,  electronic  commune iations 
signal 


Enemy  uses  electronic  signal  to  communicate  with 
other  enemy  units 
outcome:  enemy  units  communicate 

Friendly  communication  devices  get  triggered,  too. 
outcome:  friendly  knows  where  enemy  is 

Enemy  communcates  as  little  as  possible  to  avoid 
detection  *  only  important  communications  with 
parent  unit 


Economy  of  Force 
Battlefield 

Attack  one  area,  defend  other  area 

Friendly  units,  enemy  units 

Friendly  combat  power,  enemy  combat  power 


Enemy  has  two  goals 

Enemy  values  attack  more,  hence  wants  to  divert 
as  many  resources  to  attack  as  possible 
Enemy  takes  resources  from  defense  and  gives  to 
attack 

Defense  holds 
Attack  succeeds 


I 
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